[HN Gopher] A database for 2022
___________________________________________________________________
A database for 2022
Author : tosh
Score : 573 points
Date : 2022-04-01 20:33 UTC (1 days ago)
(HTM) web link (tailscale.com)
(TXT) w3m dump (tailscale.com)
| [deleted]
| rr808 wrote:
| I loved filed based h2 as well. Is there any reason in 2022 to
| use h2 over sqlite?
| jhgb wrote:
| Probably not if you can just jump to a single-file Firebird
| database, and open that one either in embedded (single-user) or
| server (multi-user) mode.
| musjleman wrote:
| So to recap their timeline: * Instead of an
| actual database use a JSON file. * Write a blog post about
| how that didn't scale. * Instead of an actual database
| hand-roll something else. * Write a blog post about how
| that didn't scale. * Instead of a database with built-in
| replication which is tailor built for key-value storage use
| SQLite with a single table containing key-value pairs and some
| fresh glue to make it replicate.
|
| I'm sorry, but what the fuck? Just writing the blog posts alone
| might have consumed more work-hours than what it would have taken
| to set a database up in the first place.
|
| Are we supposed to expect another blog post somewhere around a
| year down the road titled "SQLite didn't scale" or "The new
| library for replication that was released a few months ago
| contained a bug that took our prod down"?
| tptacek wrote:
| Do you want to know how popular systems are actually built,
| from the inside, or do you want carefully groomed triumphal
| announcements of new, perfectly-formed features?
|
| I struggle with this a lot in my writing too, but I've drawn a
| conclusion I'm sticking with: it's better to relay what the
| team is actually doing, "warts" and all. We ranked on HN a few
| weeks ago with a post about user-mode WireGuard that was
| basically a litany of engineering mistakes, and it's one of the
| things I've most enjoyed writing.
|
| Every serious product gets built the way Tailscale is
| describing. You start some components with the simplest thing
| that can reasonably work, and see how far you can take them.
| The JSON file Tailscale used to use was one of the most
| interesting and valuable things they wrote about.
|
| If you want content about how to built best-practices-compliant
| hyperscaler-grade systems, there are places to go to get that
| kind of content. But you've never had a problem with Tailscale
| that owed to their JSON file backend not scaling --- how I know
| that is, you've likely never experienced a stability problem
| with Tailscale at all, because it just doesn't break. That they
| got it to work, for a long time, with a JSON file backend
| should make you think harder about how you build things.
| musjleman wrote:
| > Do you want to know how popular systems are actually built,
| from the inside, or do you want carefully groomed triumphal
| announcements of new, perfectly-formed features?
|
| IMO both this and "announcements of new, perfectly-formed
| features" are at the same level (if not lower, because a new
| feature is still something new). Both are essentially
| marketing with no technical content that's really worth your
| time. Except that it's now the third iteration of "we did an
| obviously questionable thing and replaced it with another
| questionable solution".
|
| > it's better to relay what the team is actually doing,
| "warts" and all. We ranked on HN a few weeks ago with a post
| about user-mode WireGuard that was basically a litany of
| engineering mistakes, and it's one of the things I've most
| enjoyed writing.
|
| I can agree with that, but I wouldn't bunch your writeup into
| the same group. That's some pretty decent technical content
| about the problems that were encountered. Meanwhile in these
| writeups what were the problems? "JSON file database too
| slow", "Hand-rolled software has to be maintained in-house"?
|
| > Every serious product gets built the way Tailscale is
| describing. You start some components with the simplest thing
| that can reasonably work, and see how far you can take them.
| The JSON file Tailscale used to use was one of the most
| interesting and valuable things they wrote about.
|
| Yet they keep changing it, migrating, writing tests for the
| new code and writing blog posts. The time all these things
| take even if the size of the new code is less than 100 lines
| is substantial.
| rob_c wrote:
| Yes but userland wireguard is unique and useful. This is
| ignoring textbooks, competitors who write about their
| exploits warts and all and screams of get a more experienced
| db team to work with you... shoving it into a json screams of
| throw it at a disk and let the hardware deal with it
| lostcolony wrote:
| >> shoving it into a json screams of throw it at a disk and
| let the hardware deal with it
|
| Which...worked. For quite a long time. And required no dev
| effort. Freeing devs up to focus on other things.
|
| They did it as a POC, and it worked, better even than they
| had anticipated; why over-engineer it up front? Even when
| it turns out they are gonna need it (i.e., YAGNI no longer
| applies), they have been able to push off their "solve
| problems customers don't care about" to instead favor
| "problems customers do care about". That...seems like sound
| engineering to me?
| rob_c wrote:
| Speaking as someone who has had to fix things like
| this...
|
| When you reach capacity and it hits a screaming halt as
| the hardware capacity has been reached and performance
| doesn't collapse its completely gone.
|
| This is not about best use. Hell json file performance
| may be well within spec of the system requirements, but
| jumping up and down without perf numbers or specs is
| typical of sweep it under the carpet for a generation...
| tptacek wrote:
| That would be a good critique if it was was happening,
| but they gracefully transitioned from a JSON file(!)
| backend to etcd, and gracefully transitioned from etcd to
| sqlite. I'd say they're going to gracefully transition
| from sqlite, but I think the think people don't have
| their heads around is that it's possible they'll never
| need to do that, because of The Unreasonable
| Effectiveness of SQLite.
| cogman10 wrote:
| > Do you want to know how popular systems are actually built,
| from the inside, or do you want carefully groomed triumphal
| announcements of new, perfectly-formed features?
|
| It depends on what that tool is doing and if the thing it's
| doing is a core competency for your company.
|
| It absolutely makes sense to geek out about stuff like this
| if it's part of your core product. However, if it isn't, then
| do yourself and your engineering team a favor and pick boring
| technologies.
|
| Do you know what DB system tailscale would be using if they
| choose something boring postgres or Cassandra? Postgres or
| cassandra.
|
| The cost of not doing that is they are on their 3rd iteration
| of switching db technologies and the development effort to
| keep rediscovering the same problems.
| tptacek wrote:
| They picked (checks notes) sqlite. It's quite possibly the
| most boring choice available!
| cogman10 wrote:
| And they tacked on Litestream, which made for far more
| interesting failure modes. Hopefully they ran their
| jepson... oh wait, no they didn't really care about the
| reasons they ruled out traditional db choices.
| tptacek wrote:
| Do you understand how Litestream works? What's the Jepson
| test you'd do here, to make sure your _single, statically
| assigned leader_ isn 't conflicting with... itself? All
| Litestream does is ship WAL segments. The whole point of
| this design is not having the distributed consensus
| problems (and concomitant schema and development
| inflexibility) that etcd has.
| mrkurt wrote:
| The cost of complication is high. They're optimizing for
| simplicity. Both for low friction development and easy
| operation.
|
| sqlite is amazingly simple for local dev. Probably even
| simpler than a json file.
|
| High availability Postgres is complex to operate (you need
| a whole separate strongly consistent data store to
| coordinate it). Cassandra is also very complex.
|
| sqlite is, again, simpler. Litestream makes sqlite
| plausible for production.
| cogman10 wrote:
| > sqlite is amazingly simple for _local dev_
|
| You are buying simpler local dev with a more complicated
| and error prone production.
|
| > High availability Postgres is complex to operate
|
| High available SQLite is complicated. More complicated by
| the fact that SQLite wasn't designed at all around the
| notion of being a multi-tenant database system. Instead,
| you've got to bolt on solutions to fix the fact that you
| chose SQLite.
|
| Containers make local dev with postgres or other db
| technologies pretty slick. I have a hard time buying the
| "oh no, but then we need to run docker" because docker
| containers have added benefits often simplifying local
| dev (All your dev tools defined in the docker file. Done,
| new dev installs docker and moves from there).
|
| > sqlite is, again, simpler. Litestream makes sqlite
| plausible for production.
|
| SQLite is a fantastic tool, when it's used in the right
| location. A backend database is not the right location
| for SQLite. Litestream may make it "plausible" but it
| certainly doesn't make it a sane choice.
| yawaramin wrote:
| > You are buying simpler local dev with a more
| complicated and error prone production.
|
| How so? Any evidence for that?
|
| > More complicated by the fact that SQLite wasn't
| designed at all around the notion of being a multi-tenant
| database system.
|
| Neither was Tailscale, that's why the engineers decided
| they were a good fit. But I'm sure you know more about it
| than they do.
|
| > Containers make local dev with postgres or other db
| technologies pretty slick.
|
| No they don't. Dockerizing everything for local dev bogs
| you down to Docker's added latency for every operation,
| this is not even getting into the latencies added by
| having to periodically `docker pull`, `build` images,
| provision the disk space for the images, and maintain all
| that infrastructure over time. Compared to that, an
| SQLite database is basically nothing.
|
| > All your dev tools defined in the docker file. Done,
| new dev installs docker and moves from there
|
| Docker is not a good way to do local dev, overall. The
| entire point of local dev is speed of iteration. If you
| need a usable test environment, set one up in UAT.
|
| > A backend database is not the right location for
| SQLite.
|
| The SQLite team actually disagrees with you on this:
| https://www.sqlite.org/whentouse.html
|
| > SQLite works great as the database engine for most low
| to medium traffic websites (which is to say, most
| websites)....Systems designers report success using
| SQLite as a data store on server applications running in
| the datacenter, or in other words, using SQLite as the
| underlying storage engine for an application-specific
| database server.
|
| But again, I'm sure you know better than them.
| tptacek wrote:
| They don't have a multi-tenant database problem! This is
| the second weird thing you've said about this design (the
| other was that they'd need to run Jepson simulations for
| it). They're not a database hosting provider; they're the
| single producer and the single consumer of their
| database. They have a single writing leader, and read-
| only replicas. That's how Litestream works; it's the
| _only_ way it works.
| jbmsf wrote:
| I love Tailscale and it's true that I am not suffering from
| stability problems with its database choices.
|
| But, I do have challenges working with their ACL structure --
| which is backed by their database -- and specifically would
| love to be able to update users/groups separately from tag
| groups and both of those separately from ACL rules and I
| can't help but wonder if the fact that this feature doesn't
| exist relates to the fact that the database doesn't normalize
| these concepts into separate "tables" separated by "foreign
| keys".
| bradfitz wrote:
| Tailscale engineer here.
|
| Unrelated.
|
| We just haven't worked on it much. A bunch of other stuff
| has been a higher priority. But better ACL management will
| happen.
| structural wrote:
| This is great to hear. We've been happy users for a
| couple of years, and the device tagging support has come
| a long way (and is a big enabler for sensible ACLs), but
| trying to use Tailscale's ACLs in general has been our
| biggest pain point with the product.
| dimgl wrote:
| I have nothing to add to this discussion, but I wanted to add
| that I love this comment. Sometimes we have to take a step
| back and really think about why we're making all of these
| choices and picking certain tools for a job.
| Dylan16807 wrote:
| Okay, but this _is_ a database now.
|
| Is there a reason the replication needs to be built-in?
| [deleted]
| artificialLimbs wrote:
| I thought replication was useful for backups or testing.
| Maybe other things. edit: I know practically nothing about
| db.
| musjleman wrote:
| Sorry, I probably didn't phrase it very clearly. I don't
| think there is anything wrong with needing some external help
| for that, but the combination of their choices is IMO a bit
| bizarre: * SQLite which I mostly see as an
| embeddable database for applications, used on a server.
| * Single table with key-value pairs. * Having the tool
| for replication be a quite young project that's in active
| development.
| ripley12 wrote:
| > SQLite which I mostly see as an embeddable database for
| applications
|
| This is common but incorrect view of SQLite. It's able to
| handle many server-side workloads, as long as you're aware
| of its limitations (and the Tailscale team certainly are).
|
| https://www.sqlite.org/whentouse.html
| tptacek wrote:
| Those limitations are also the limitations of the backing
| stores of many distributed state solutions; you just
| don't see them because they're hidden behind complicated
| APIs.
| tptacek wrote:
| Plenty of people use sqlite on servers.
|
| They have a single table with KV pairs right now as an
| artifact of having used the "database with built-in
| replication which is tailor built for key-value storage"
| you complain that they're moving away from. The point of
| the blog post is that they're doing that in part to move
| away from the single table of KV pairs.
|
| They're not "sharding".
| [deleted]
| dan-robertson wrote:
| Doesn't the blog post explain why they chose this step rather
| than doing the thing you suggest? In particular I don't see how
| this comment really addresses any of their arguments.
|
| And if they got on ok with the json file, maybe you should
| update towards the features of a complicated RDBMS not being
| worth the cost?
| cogman10 wrote:
| No, it actually doesn't. The blog post explains "Hey, we've
| chosen a bunch of crazy ways to store data on the backend. We
| looked at non-crazy ways, but decided instead it'd be more
| fun to rsync SQLite data around because SQLite is my favorite
| DB!"
|
| > MySQL (or PostgreSQL) would come next. I'm not particularly
| familiar with anything MySQL post 1998, but I'm sure it would
| work. The HA story for open source databases is somewhat
| surprising, though: you can either have traditional lagging
| replicas, or commit to no-primary-replica clusters that have
| very surprising transaction semantics. I wasn't excited about
| trying to design a stable API or good network graph
| calculations on top of those semantics. CockroachDB looked
| very promising, and indeed still does! But it's relatively
| new for a database and I was a little concerned about getting
| attached to features in a fresh DBMS that would be hard to
| migrate away from if we needed to.
|
| Seriously.. consider this.. they are worried about the
| transaction semantics of using db replicas and instead CHOSE
| SQL LITE!
| bccdee wrote:
| Sqlite replication via litestream has bascially no
| overhead. It's a great choice if your database only needs
| to service one client.
|
| It sounds like you're essentially arguing that people
| should always use the most robust data store available
| regardless of cost or requirements. Their needs were served
| just fine by a json file for quite some time -- if that's
| all you need, then maintaining a full Posgres cluster is
| the "crazy" solution actually.
| tptacek wrote:
| And?
|
| (It's sqlite, or SQLite if you're being pedantic; it's not
| "SQL LITE").
| bradfitz wrote:
| Tailscale employee here.
|
| Our database needs are tiny, as explained in the earlier post.
| So we optimize for things like: "can we run all our tests
| quickly and easily in many environments without containers and
| VMs?" All three of our storage schemes have had that property.
|
| We have MySQL and PostgreSQL veterans on the team. We know
| those options well.
| cogman10 wrote:
| > without containers
|
| Why?
|
| The official docker postgres package weighs in at 100MB.
| Assuming you standardized on that, then every environment
| would end up having a fresh postgres image just waiting to be
| startup. Meaning, the actual cost is the memory for the
| server and startup time, not the image itself.
|
| Mount the data on tmpfs and you can startup and setup a db in
| very little time (I know, because that's what we do).
|
| You might be able to eek out a bit more performance by hand
| tuning test files... but.. to what end?
| tptacek wrote:
| I can't speak for Tailscale, but our whole raison d'etre is
| running containers for people, and almost none of our
| development environments are containerized either.
| Containers (and remote dev environments) create friction.
| They're useful tools when that friction is unavoidable
| anyways, but when it isn't --- and you can design systems
| with that being a goal --- it makes sense to take full
| advantage of that.
| dgb23 wrote:
| Have a look at this headline:
|
| > Zero config VPN. Installs on any device in minutes,
| manages firewall rules for you, and works from anywhere.
|
| I don't know how their product works, but my intuition is
| that your suggestions aren't feasible.
| cogman10 wrote:
| I suggest reading the article. The reason they haven't
| chosen "boring" tech has nothing to do with the clients
| running their software and everything to do with their
| internal development experiences.
|
| > MySQL (or PostgreSQL) would come next. I'm not
| particularly familiar with anything MySQL post 1998, but
| I'm sure it would work. The HA story for open source
| databases is somewhat surprising, though: you can either
| have traditional lagging replicas, or commit to no-
| primary-replica clusters that have very surprising
| transaction semantics. I wasn't excited about trying to
| design a stable API or good network graph calculations on
| top of those semantics. CockroachDB looked very
| promising, and indeed still does! But it's relatively new
| for a database and I was a little concerned about getting
| attached to features in a fresh DBMS that would be hard
| to migrate away from if we needed to.
|
| The reason they are ruling out products isn't "this won't
| work where it runs" its "I don't like this". They talk
| about worrying about network graphs and proceed to use
| SQLite and copy garbage around.
| tptacek wrote:
| If you're dinging them for not using Cockroach, they
| explained why they're not using Cockroach in the exact
| paragraph you quoted: if they use Cockroach, they're
| committing to the scaling and distribution system that
| Cockroach provides, which might or might not be a good
| fit for them 2 years from now. If it isn't a good fit,
| they're stuck with a huge engineering bill to get
| themselves out of Cockroach.
|
| We're moving towards sqlite for things for the same
| reason. sqlite is simple to reason about. The distributed
| state problem not simple, but our distributed state
| problem is not Tailscale's and theirs isn't yours; every
| distributed state problem is unhappy in its own way. As
| they work out the contours of their specific distributed
| state problem, sqlite isn't the component that's going to
| break down.
| cogman10 wrote:
| > If it isn't a good fit, they're stuck with a huge
| engineering bill to get themselves out of Cockroach.
|
| Similar to the current engineering bills they are paying
| because they switched from a json file -> etcd -> sqlite?
|
| Seems like the cost of switching isn't really an issue
| for them since they've pulled that lever multiple times
| now.
| tptacek wrote:
| It does seem that way. Keep thinking along those lines.
| You're almost there!
| jrockway wrote:
| I write applications that use Postgres, and don't use
| Docker for testing. I just create a database based on the
| name of the test case, delete it, create it, explode the
| schema in there, and run the tests. The reason is because I
| don't want to pay Postgres' startup costs every time I run
| the tests. Tests are run thousands of times a day, while
| you're in the mindset to rapidly iterate. Any delay is
| unacceptable.
|
| At some point, migrations were taking too long, so I just
| started caching the database schema. I have a go generate
| thingie that creates a fresh database, applies all the
| migrations, and dumps the schema to a file that gets
| checked in with the code. (CI checks that the migrations
| actually result in the database state that's checked in.
| Yes, there is something that regexes out the hostname from
| the sql file, since Postgres dumps that in there by default
| and that results in spurious diffs.)
|
| I have learned that people often put up with really sub-
| optimal developer workflows. My test is that you should be
| able to add "print hello world" to the top of your main
| function, and see that printed on your screen in less than
| 5 seconds. (Yes, even if your application is complicated. I
| have a hacked up copy of K8s that starts that quickly, so
| that you can write an app that deeply integrates with K8s
| and not have to reuse a cluster between tests, or pay a
| long cluster creation cost. All obstacles must be removed!)
| If it takes longer than that, you are just throwing your
| developer's salaries into the toilet. (If it takes 6
| seconds, you open up HN, and there's an hour gone!)
| [deleted]
| epolanski wrote:
| > we run all our tests quickly and easily in many
| environments without containers and VM
|
| How's that an important metric to optimize for? Did you ever
| benchmark the tests with other solutions?
| jen20 wrote:
| In what world would "make dev environments better" _not_ be
| an important thing?
| mrkurt wrote:
| I believe "quickly" means "quick for a human to do" in this
| case.
| srameshc wrote:
| Why not ? You can be a spectator to it watch them succeed or
| fail.
| [deleted]
| [deleted]
| VWWHFSfQ wrote:
| It seems like they're _really_ struggling with is. It's kind of
| entertaining to watch though
|
| edit: also don't forget these blog posts are content marketing
| for their product. that's really the point anyway
| yawaramin wrote:
| If from your perspective, a super-successful VPN product is
| 'really struggling', you must have incredibly high standards!
| We all hope to hear more about your successes, I'm sure.
| Aeolun wrote:
| > In the first place
|
| Yes, but ongoing maintenance isn't free.
|
| If anything, this whole DB funkiness indicates to me just how
| nice a product you can build despite it.
| ithkuil wrote:
| > This gives you near real time backups (*or with a couple deft
| modifications, lets your app block at critical sections until the
| backup is done*)
|
| anyone has an example of how to do this?
| ramenmeal wrote:
| I think I'm missing some context. Using a text file and using
| etcd as a DB for a production system seems like a terrible
| engineering decision. It seems like something you'd do as a proof
| of concept or side project. It's interesting that they're
| blogging about this, as if they're proud of it. I guess I'm just
| missing the point. This is their 3rd DB Migration, something that
| I prefer to avoid at all costs. I guess they have different
| priorities and values. But I'm just confused, why would an
| engineer want to join a company that is making these decisions?
| Why would the company want their users to know about these
| decisions? They started using etcd knowing that he would become a
| bottle neck, that should have been a non-starter, right?
| kevin_nisbet wrote:
| > But I'm just confused, why would an engineer want to join a
| company that is making these decisions? Why would the company
| want their users to know about these decisions?
|
| This might actually be a really good filter for which types of
| engineers are good fit for their company. Because my read was
| very different than yours, as I looked at the decision lineage
| and thought to myself I could see myself making every one of
| those decisions. I of course can't say I would've with the same
| problems and constraints they had, just that I know I've
| considered similar solutions.
|
| The big difference might be, on the third iteration I tend to
| land on embedding raft directly into the backend, because lots
| of the problems I look at tend to benefit from a in-order log
| that is fully replicated to a quorum of servers. But based on
| the post and efficiency they're going for backing up the WAL on
| sqlite may make alot of sense.
|
| Actually litestream looks like it might have some fairly
| similar properties, so might be something else I'd like to add
| to my toolbox. Although I wonder what sort of replication lag
| there might be to replicas when using s3 as the stable store.
| benbjohnson wrote:
| Litestream author here. The default configuration is to
| replicate any new changes every second to S3. This is mainly
| a cost issue though. S3 doesn't charge for ingress bandwidth
| but it does charge $0.000005 per PUT request. If you have a
| non-stop, continuous stream of writes against your database
| then that's about $12/mo. In practice, most people don't have
| a continuous stream of writes. Michael Lynch, for example,
| wrote a post about how he pays $0.03/month for his Litestream
| replication[1].
|
| I've worked with Raft a lot in my career. I wrote the
| original etcd Raft implementation (sorry!) [2] and even wrote
| a visualization explaining Raft[3]. It's a great tool when
| you need really high durability guarantees but I'm not
| convinced it gives you better availability guarantees than a
| single node that can quickly be recovered.
|
| [1]: https://mtlynch.io/litestream/ [2]:
| https://github.com/goraft/raft [3]:
| http://thesecretlivesofdata.com/raft/
| lmeyerov wrote:
| There's a world of a difference of someone doing that for a
| Google scale thing with big teams supporting it, or say a
| core part of making the best wireguard product that only they
| can do, vs a system that could have been postgres and now
| when the authors leave for their own sqlite-by-mail startup
| or are partying at SREConf for the 'our n+1th attempt to not
| do postgres' and an outage causes an unnecessary and
| ungoogleable shitshow for their colleagues.
|
| A good engineer can build all sorts of surprising things. A
| senior one knows why they should reserve that for only the
| special bits. I like reading the first two types of articles
| -- we frequently give tech talks ourselves -- but for making
| boring systems weird, I get flashbacks of
| consultantware/resumeware/NIH that took ages to weed out. I
| try to give talks on say GPUs for weirdly shaped problems,
| not on why we celebrated when we finally had time to rip out
| mongo from where it didn't belong :)
| billywhizz wrote:
| the live replication feature litestream is implementing won't
| be using s3 - it will be direct node to node replication.
| hasn't been released yet and currently s3 is used for a cold
| backup that can be quickly restored to a fresh node as far as
| i know. https://github.com/benbjohnson/litestream/issues/8
| bradfitz wrote:
| Tailscale engineer here.
|
| > Using a text file and ... > It seems like something you'd do
| as a proof of concept
|
| I mean, using a text file for the proof of concept is exactly
| what happened. And then it grew too quickly and had to get off
| of it eventually, but we always knew that. We were just amazed
| how long that worked. It survived much longer than we'd
| thought.
| mardifoufs wrote:
| matthewmacleod wrote:
| _Using a text file and using etcd as a DB for a production
| system seems like a terrible engineering decision_
|
| I'd say this could easily be a great engineering decision if it
| meets the needs of an organisation in the moment. The thing
| with any engineering decision is that it involves a bunch of
| trade-offs - doing something that you know will hit a scaling
| wall can be worth it, if it can get you moving faster. There
| will never be a perfect solution, and in these decisions I find
| that simplicity is often a _massively_ under-rated benefit.
|
| I'd even venture to say that I've seen more organisational
| failures as a result of optimistic over-engineering than I have
| optimistic under-engineering. That includes many of my own
| projects - ones where I've gone "oh I guess I'll do it properly
| this time" and then found myself six months down the line
| throwing out a ton of code, schema, or infrastructure I spent
| ages on because some requirements changed or some assumptions
| didn't pan out. It's hard to remember that lesson sometimes and
| I've been doing this for 15 years.
|
| It's like a company I used to work for where the whole product
| was driven by a massive semi-structured graph database - but
| instead of being implemented like that it was a big ball of
| MongoDB JSON documents glued together with PHP by someone who
| was more of a domain expert than an engineer. Often pretty
| sketchy to work on, but an engineering decision that was a
| success - in the sense that it allowed that product to exist at
| all where it otherwise might not have.
|
| _why would an engineer want to join a company that is making
| these decisions? Why would the company want their users to know
| about these decisions?_
|
| Sign me up as both an engineer and a user. It is
| extraordinarily refreshing to hear "we tried a bunch of
| different stuff, some of it was dumb, and this is where we're
| at now."
| hokus_p2 wrote:
| Aeolun wrote:
| > why would an engineer want to join a company that is making
| these decisions?
|
| Hmm, I can't quite articulate why, but I would. Something about
| having the courage to be different.
| fiddlerwoaroof wrote:
| Yeah, it's perfectly reasonable to use a text file instead of
| a database until you find evidence that you need a "real"
| database.
| dave78 wrote:
| I would actually love to work at a company like that. I've
| been a professional developer for over 20 years now and at
| least in my experience, too much unneeded complexity has been
| the root cause of so many defects and product failures. I'm
| constantly fighting battles to keep things simple. All the
| young devs on my team want to use every fancy new technology
| that passes by, but I just want things to work for my
| customers with the smallest amount of downtime and late night
| pages.
|
| If a single JSON file worked at the time, then that's clearly
| all they needed. It certainly hasn't seemed to limit their
| success.
| billywhizz wrote:
| > All the young devs on my team want to use every fancy new
| technology that passes by, but I just want things to work
| for my customers with the smallest amount of downtime and
| late night pages.
|
| i feel your pain dave. resume driven development and FAANG
| envy seem to be the dominant characteristics of far too
| many folks in the industry these days.
| rudasn wrote:
| This is an April fool's joke, right?
|
| Edit: no it's not..
|
| > Footnote: coworkers point out it's April Fool's today and
| request that I clarify this isn't a joke. Joke's on them: every
| day's April Fool's in the Tailscale Database Engineering
| department.
| [deleted]
| [deleted]
| bradfitz wrote:
| I suggested that footnote text just for these comments. :)
| thrower123 wrote:
| I'm getting half-remembered flashbacks to some of the
| distributed, self-replicating Lotus Notes applications I worked
| on very early in my career
| SodiumMerchant0 wrote:
| ctopher7 wrote:
| a little bit oot, but just installed tailscale on my dev stack
| (ec2, bare metal at home and my macbook air). mindblown by its
| simplicity and "it just works". Use it for logging purposes (ELK
| is too heavy for free tier ec2) and CI/CD (again, too heavy for
| my free tier ec2)
| dahfizz wrote:
| This should be a new logical fallacy: appeal to current year.
| Every time I hear someone say "it's $CURRENT_YEAR, we obviously
| need to implement $PARTISAN_POLICY" I wonder about what fringe
| politics will be "obviously correct" in 3022.
| [deleted]
| AnimalMuppet wrote:
| Seconded. It's a logical fallacy because they are using the
| year as _proof_ that we should adopt their policy. Except the
| year only proves what year it is - _nothing_ else.
| billywhizz wrote:
| oh boy. is nobody allowed to make a little joke anymore. you
| should take yourself _a lot_ less seriously my dude.
| AnimalMuppet wrote:
| First: I am not "my dude" to you.
|
| And, completely apart from the database, it's a really
| valid point - not a joke at all.
|
| When someone says, "It's 2022. Why are we still arguing
| about X?" or some such, it means that they haven't been
| able to persuade the rest of the world about X on the
| merits of their argument, and so they want to short-circuit
| the argument by appearing "modern" or "up to date" or "on
| the right side of history". They want to win by persuading
| the other side to give up in embarrassment, not by virtue
| of the quality of their position. (In their own eyes, their
| position is probably self-evidently the only right and
| moral one, but that doesn't tell us much.) It's a way of
| trying to win by default, and it deserves nothing other
| than being regarded as "you lose by default".
| billywhizz wrote:
| hu3 wrote:
| not everything is a joke. and some jokes are not funny.
|
| specially ad hominem criticism.
| tedunangst wrote:
| Nothing in this post claims that the year intrinsically makes
| this the correct choice. It is the database for 2022 because it
| is now 2022 and this is the database they are now using.
| maximilianroos wrote:
| I see a dozen comments here, all critiquing their proposal.
| Surely this is April Fools?
|
| > I'm sorry, but what the fuck? Just writing the blog posts alone
| might have consumed more work-hours than what it would have taken
| to set a database up in the first place.
|
| > Using a text file and using etcd as a DB for a production
| system seems like a terrible engineering decision. It seems like
| something you'd do as a proof of concept or side project. It's
| interesting that they're blogging about this, as if they're proud
| of it.
|
| > This is their 3rd DB Migration, something that I prefer to
| avoid at all costs. I guess they have different priorities and
| values. But I'm just confused, why would an engineer want to join
| a company that is making these decisions?
| jjeaff wrote:
| They put an update at the bottom of the article clarifying that
| it is not, in fact, a joke.
| kall wrote:
| At first I was also a little confused by this architecture. I
| think that was because I was missing a clear picture of what they
| want to do. I think it is:
|
| - one leader that can write to the db
|
| - a ton of other hosts that can read from the "streamed" db and
| want to do so often/fast
|
| - hosts send their writes (if they have any?) to the leader via
| some api call (?)
|
| Is that right? If so, the solution makes a ton of sense. "Just
| use pg/MySQL" in that case would mean hundreds (thousands?) of
| read replicas, which doesn't sound fun, or every read over the
| network.
| tptacek wrote:
| It's replacing etcd, which has essentially that model.
| Razengan wrote:
| Who would use it if it's just for one year?
| endisneigh wrote:
| I'm so desperate for a SQL database where I can just put it on a
| bunch of commodity hardware via Docker, connect them up and never
| worry about this again. Ideally it'd monitor my queries and
| create indexes for me. pleeeeeeaaaaaaaaaaaassssseeeeeeeeee
|
| /dream
|
| FoundationDB is similar but you have to do so much yourself. It's
| more or less what I'm describing for a key value store though.
| Not sure why it's not more popular.
| btgeekboy wrote:
| FoundationDB is an Apple thing, isn't it? Might explain why. It
| feels kinda like they've put it out there and it just hasn't
| had anyone with a marketing budget to push it along.
| database_guy wrote:
| It started as a proprietary database. Apple acquired it and
| then open sourced it eventually.
| tapirl wrote:
| Didn't it start as an open source database? then apple
| acquired it and made it close sourced, then open sourced it
| again after several years.
|
| It is weird that this DB has a good reputation which
| doesn't match its popularity.
| jitl wrote:
| FoundationDB was the first publicly available distributed
| transactional KV on the market that I can recall. It
| wasn't "open" though - always proprietary. They generated
| a lot of hype and good press for being this NoSQL DB that
| was all about ACID correctness in the age of "mongo is
| webscale (by writing to /dev/null)". They had a demo
| showing impressive throughout on a low power cluster with
| no data loss during a power cycle. I think that despite
| generally impressive tech they had some trouble selling
| but did end up selling the company to Apple.
| jen20 wrote:
| No, it was not open source prior to Apple releasing it as
| Apache 2.0 in April of 2018.
| astrange wrote:
| The source was licensed to some people but it wasn't open
| source IIRC.
| [deleted]
| odonnellryan wrote:
| You probably don't really want that. But not in the way that
| people didn't want WoW classic.
| bobnamob wrote:
| Can you elaborate on why you say this?
| database_guy wrote:
| I lamented this exact situation in a post I made last week. I'm
| with you on this completely.
| jitl wrote:
| I am hoping EdgeDB heads this direction to some extent.
| Nican wrote:
| I am having quite a good time with CockroachDB. It has been
| mostly set-and-forget, and it auto-balances, and does rolling
| upgrades pretty smoothly.
| marginalia_nu wrote:
| Solving performance by mindlessly adding indexes works great as
| long as you don't have all too much data and have endless RAM.
|
| The index backing the unique constraint on the URL table for my
| search engine is around 23 Gb. The entire server has 128 Gb of
| RAM, for comparison.
| tptacek wrote:
| This thread has (at least at the moment) serious Bob Martin
| Sudoku Solver energy to it. Tailscale has solved an infamously
| complicated problem using, for the most part, simple tools.
| They're not just successful; they're remarkably successful,
| spookily successful, upsettingly successful.
|
| Consider whether the secret sauce here might not be _au courant_
| database choices, but rather something much harder for random
| teams to pick up and run with: just straight up good design and
| programming.
|
| This is a thing that happens in other fields, too; some of the
| hardest dishes to cook seem extremely simple and have few
| ingredients. They're hard because there's no place to hide: your
| technique has to be flawless or they don't work. A famous
| entrance exam for new cooks at fine dining restaurants is "make
| an omelet". Tailscale is, thankfully (and, to me, irritatingly),
| more like an omelet than a truffled hash.
| dastbe wrote:
| nobody goes to these restaurants for their omelets, though at
| the same time nobody should care how tailscale actually
| implements things so long as it works well.
|
| imo they're not solving a notoriously hard problem, they're
| realizing that they have the tiniest version of this problem
| (and that will most likely continue to be true). when that's
| the case and you want to prioritize stuff that moves the needle
| like dev velocity, why do anything less simple? getting hung up
| on building scalable solutions far beyond what you need is
| somewhere between a fool and a big co employees game. far too
| often people introduce complexity way before it's needed, and
| that almost always slows down your devs ability to actually fix
| bugs and make the product more reliable.
| [deleted]
| tptacek wrote:
| They don't, but there are other "finesse" dishes that people
| do in fact go to restaurants for that are notable for their
| simplicity, and the fact that you can only pull them off if
| you have flawless technique. Japanese, in particular, is
| notorious for this.
| jimbokun wrote:
| There's a manga about a food critic (Oishinbou [1]), where
| he takes another food critic to a hole in the wall Tokyo
| restaurant as an example of elite cooking. The chef-owner
| brings him his meal.
|
| Which is just a piece of cooked fish, rice, and miso soup.
| The critic is first almost insulted by such a simple meal.
| But then realizes the fish, rice and soup are prepared to
| perfection, and is amazed at the chef's skill.
|
| [1] https://en.wikipedia.org/wiki/Oishinbo
| mbreese wrote:
| Which is basically also the ending of the Pixar film
| Ratatouille. The protagonist wows a critic with a simple
| dish prepared to perfection. It's a common trope, but
| also true. "Simple" food prepared well can be much better
| than a complex dish with many flavors.
|
| Similarly, I've also heard (probably from a show like Top
| Chef, so YMMV) things like a salad or soup are also quick
| ways to differentiate chefs. In this case, it's less
| about technique and more about identifying and mixing
| flavors.
|
| It works because there is no place to hide. You either
| have good flavor or not.
| hparadiz wrote:
| +1 on good design and programming.
|
| My go to language of choice is PHP/MySQL. I've never not been
| able to scale a project. At one point while working at Comcast
| I prototyped a system for every error code from every cable set
| top box (150 million) in the country on a MySQL server instance
| running on a MacBook Pro pulling data from Splunk in real time.
| All so I could generate some png charts and embed them in
| Slack.
|
| I know there are limits to MySQL but so far in my career I
| haven't had any projects hit those limits because hardware has
| been getting better faster.
| BonoboIO wrote:
| But my Kubernetes Cluster on AWS can handles this for only
| 30.000$ a month
|
| That Comcast prototype sounds amazing. PHP and MySQL can
| really bring you very far, i use it for nearly everything and
| it is so incredible easy to develop and "deploy" new
| versions. I use propel as ORM and it is very close to the DB
| and I can change schemes fast and just let the autoupdate
| handle the rest.
| cbg0 wrote:
| > But my Kubernetes Cluster on AWS can handles this for
| only 30.000$ a month
|
| A k8s cluster for something like that wouldn't cost
| anywhere that amount, plus you would get the advantage of
| being able to deploy your app in multiple availability
| zones. If you don't need k8s features, there are definitely
| simpler options that will give you redundancy and won't
| break the bank.
| hparadiz wrote:
| I only used a MacBook cause it was faster to ask IT for
| one than to have my ticket processed and approved by the
| cloud and security team which both required a write up on
| why I need a vm and what justification I have for needing
| it. Meanwhile IT is like "here you go".
| oogali wrote:
| This comment makes me unreasonably giddy.
|
| The replies are focused on MySQL (and to a lesser extent,
| PHP). None of that matters.
|
| The hero here is the logging system serving as seekable
| stream of collected events. (I'm trying hard here to not say
| K-fka).
|
| By consuming events from that source and being able to
| perform time-based seeking, you've offloaded (or dare I say,
| eliminated) your scaling pressures.
|
| Even if you commit the "sins" of no indexes on heavy queries,
| using MyISAM, or building a giant, memory-intensive
| associative array... none of it matters.
|
| You can choose to wipe your local dataset, start again in a
| different language, switch to another data store model, and
| none of your downstream consumers will know nor care (unless
| you blog about it).
|
| *This* type of design, that prioritizes consumer-facing
| resiliency and continuity, is what I look for in systems that
| scale.
| ransom1538 wrote:
| "know there are limits to MySQL"
|
| I have gone past mysql limits a few times. Will you go beyond
| its capability is a simple measure. Can your hardware write
| fast enough to keep up? If your hardware maxes out writing to
| disk 200 megs per second, welp mysql maxes out at 200 megs
| per second. If you need 400 megs per second of writing, well
| you need two instances that can write independently. That is
| when you are fucked. At that point you get into sharding* and
| doing whacky things mysql isn't meant todo. Reading things
| from disk, mysql's replication scales just fine.
| vaughandroid wrote:
| Please could someone explain the "Bob Martin Sudoku Solver
| energy" reference?
| diarrhea wrote:
| All I found: https://news.ycombinator.com/item?id=15441323
| ransom1538 wrote:
| One of the largest forums online in 2003 was was ezboard. A
| billion page views per month. Back then - insane - well that
| was beyond insane. I started work there as a young idiot. "What
| db you guys using, mysql?...." long pause... "We use a file
| system." I dug in. Yep. A fucking file system. Files allowed:
| simple backups, impossible speed, obvious cache systems,
| partitioning.
|
| Later on i met the roomate of the author of memcache. Few
| hundred lines of C. Completely changed the game how caching
| systems worked. Years later a super senior FB engineer
| explained to me over a beer - "we basically use mysql as a file
| store, everything else is in memcache." Good ideas are stupid
| simple.
| tptacek wrote:
| The guy who wrote memcache is on this thread, I think?
| ransom1538 wrote:
| bradfitz lol he is.
| hashhar wrote:
| And he even works at Tailscale now.
| carapace wrote:
| (Not to namedrop, but, uh) I know apenwarr glancingly IRL and
| he's amazing. A force of nature. When people talk about "10x"
| programmers I think of him.
| ignoramous wrote:
| > _This thread has (at least at the moment) serious Bob Martin
| Sudoku Solver energy to it._
|
| To the forum's credit, the top comment (by u/judofyr) the last
| time tailscale blogged about their unusal db setup, got it
| exactly right: https://news.ycombinator.com/item?id=25768042
|
| > _Interesting choice of technology, but [tailscale] didn 't
| completely convince me to why this is better than just using
| SQLite or PostgreSQL with a lagging replica._
|
| > _In particular [tailscale has] designed a very complicated
| system: Operationally you need an etcd cluster and a tailetc
| cluster. Code-wise you now have to maintain your own
| transaction-aware caching layer on top of etcd. That 's quite a
| brave task considering how many databases fail at Jepsen._
|
| > _Considering that high-availability was not a requirement and
| that the main problem with the previous solution was
| performance ( "writes went from nearly a second (sometimes
| worse!) to milliseconds") it looks like a simple server with
| SQLite + some indexes could have gotten [tailscale] quite far._
| tptacek wrote:
| I did commit the cardinal sin of HN metacommentary and
| referred to the state of the thread at a point in time as if
| it was a durable property of the thread, which was a mistake
| I own and take full responsibility for.
| reilly3000 wrote:
| If anything, this ^^^ is how you win the long game. History
| will surface itself when you make assertions. Gracefully
| revocability scales.
| 0des wrote:
| At least he's not the Dropbox guy
| Issaclabs wrote:
| What does that mean? Can you explain?
| Operyl wrote:
| This comment trivializing Dropbox:
| https://news.ycombinator.com/item?id=9224
| pooper wrote:
| I for one do not think the comments trivialized anything.
| I see these as inevitable questions that a leader like
| Drew Houston should be prepared to answer at some point.
| Think of it this way, would you disqualify me if you were
| hiring and I asked a similar question at the interview?
|
| I think the underlying actual question for Dropbox to
| ponder (but perhaps not answer publicly) was "what is
| your moat?" without the business development language and
| at least for me that remains an open question.
|
| I remember I was in college when Dropbox started and I
| thought Dropbox was silly to offer more storage for
| inviting more users. It was easy to approach people and
| ask them to try something in college. I think I almost
| maxed out my invites. I remember feeling I was taking
| Dropbox to the cleaners...
|
| One activity question for HN: why do you think Dropbox
| continues to exist today while copycats like Barracuda
| Copy have failed?
| belter wrote:
| That comment is unfairly judged, taking into account how
| much they used AWS instead of their own solution.
|
| "...Half-a-billion people stored files on Dropbox. Well,
| sort of. Really, the files were in Amazon's cloud. .."
| [1]
|
| [1] "The Epic Story of Dropbox's Exodus From the Amazon
| Cloud Empire"
|
| https://www.wired.com/2016/03/epic-story-dropboxs-exodus-
| ama...
| emteycz wrote:
| Why can't a company use cloud? It's not like there's
| Amazonbox client I could install now and get the same
| service. Dropbox never was about the storage itself, IMHO
| - it was about the ease of usage. There were enough
| options to get storage - cloud or non-cloud - when it
| came out, but none of them had the easy folder-based sync
| and sharing (public folder) of Dropbox.
| belter wrote:
| It's not if the company uses cloud or not. It's if they
| are solving the hard problems or instead, outsourcing the
| technical challenges to their cloud provider while
| charging the user a non justifiable premium for the
| intuitive user interface :-) So it's about the eventual
| value add and the business case.
|
| The challenging part is to provide the back-end. Get
| massive storage, with proper checksum integrity, at a
| cost effective value and with flawless security.
|
| A single command (and minor variations around it):
|
| aws s3 sync <name_source_dir> s3://<name_target_bucket>
|
| get's you all that. It can be easily mapped to an
| intuitive user interface by a software development
| boutique in weeks.
|
| And because that is the difficult part, it seems for the
| first 8 years of their history that was solved for them
| by AWS, according to their own reported history.
|
| Sharing a folder or file from Dropbox is no more
| intuitive than using the S3 console. Looking at the very
| colorful history of security issues around Dropbox [1]
| it's certainly not safer.
|
| So why would a user pay more for storage costs than with
| a cloud provider, with notoriously less security? Does a
| tiny layer on top one or more cloud providers justify the
| add-on value for the user or the current valuation of 9
| Billion dollars?
|
| [1] "Criticism of Dropbox":
| https://en.wikipedia.org/wiki/Criticism_of_Dropbox
| reilly3000 wrote:
| And for what its worth, systems like minio have emerged
| due to that commercial innovation. AWS has made a lot of
| investment in that system and will earn perpetual
| annuities from it, but by creating a defacto protocol we
| now get many choices that use its API. Its interesting
| because AWS never was benevolent with their S3 code, they
| just created a novel and rigorous offering that became
| worthy of emulation. Dropbox really hasn't done any
| equivalent contribution. If anything, the open source
| projects that have emerged to offer self-hosted
| alternatives to it are more feature-complete and
| extensible- such as nextcloud/owncloud, pydio, syncthing,
| etc.
|
| side note: I'm filled with regret for every document that
| I authored on Dropbox Paper. It didn't last for long, but
| there was a time that it was the official wiki for a
| company I worked at. Its not impossible to export, but
| never with full-fidelity.
| zimpenfish wrote:
| > Sharing a folder or file from Dropbox is no more
| intuitive than using the S3 console.
|
| I've not really used the S3 console but, at least on
| macOS, sharing a file from Dropbox is as simple as
| "right-click on the file in Finder, Share..." then enter
| an email address or create a share link from the popup.
| Or I can right-click and "Copy Dropbox Link" if the
| direct link is good enough.
|
| Is the S3 console simpler than a right-click?
| belter wrote:
| Even easier and safer, create a signed url that expires
| after 5 min...Send it to the user...
|
| aws s3 presign s3://awsexamplebucket/test2.txt --expires-
| in 300
| danenania wrote:
| The median Dropbox user doesn't know what s3 or is what
| 'console' means.
| emteycz wrote:
| Their fundament is UX, not technology. Their technology
| could be whatever and users wouldn't care - the users
| care about syncing their files easily. Few geeks might
| flame about it on a tech forum, but nothing more.
|
| Not that I like Dropbox too much. I'm going to switch to
| my self hosted solution. But it's certainly not as easy
| as using Dropbox, even as someone who uses AWS since 2012
| frequently.
| stavros wrote:
| The Dropbox guy admitted he was wrong too, but we won't
| let him live it down. At least we don't remember his
| name.
| techdragon wrote:
| This and the "no Wi-Fi less space than a Nomad" comments
| on slash dot about the iPod are my favourite examples of
| "you don't get consumers" comments.
| ignoramous wrote:
| to quote u/InGoodFaith
| (news.ycombinator.com/item?id=29555814):
|
| > _Wouldn 't also be HN without the misinterpretation of
| BrandonM's response. Here is dang's comment about that
| topic/meme:
| https://news.ycombinator.com/item?id=29178442_
| 0des wrote:
| As someone who was there to witness it when it happened,
| that is not the impression I got at all, and I think Dang
| may have taken the more humble interpretation in that
| remark.
| stavros wrote:
| You don't need to have been there, though, as anyone can
| read the perfectly-preserved exchange even now.
| recuter wrote:
| And some things that should not have been forgotten were
| lost. History became legend. Legend became myth...
| jldugger wrote:
| > serious Bob Martin Sudoku Solver energy to it
|
| You mean Ron Jeffries?
| morelisp wrote:
| The Bob Martin Sudoku Solver is the same thing but twice as
| expensive and replaced what little self-awareness it had with
| unnecessary sexual innuendo.
| tptacek wrote:
| I might!
| jrockway wrote:
| People seem really stressed that they've changed databases
| twice before. Their service never went down and they didn't
| stop delivering features while all this was happening, so I
| don't see the big deal. You pick what works at the time, and
| incrementally evolve as problems and solutions become apparent.
|
| This article could have easily been "why we were down for a day
| while some auto-vacuum setting was broken." Then everyone would
| be like "why didn't you just scp a json file between machines"
| ;)
| tptacek wrote:
| Another comment on this thread tried to dunk on them, saying
| that they'd done this so many times before (JSON to etcd,
| etcd to sqlite) that they must be getting really good at it.
| That comment struck me as so close to understanding what
| they're doing right here, it was painful.
| jrockway wrote:
| Yeah. Maybe people are overestimating how expensive this
| kind of change is. We can all imagine being part of a
| larger org, where one day an All Hands appears on your
| calendar, and some exec you've never heard of proclaims the
| beginning of a 2 year multi-team database migration
| project. So now instead of making nifty software, you are
| just in 40 hours of meetings a week fighting over small
| details on this migration nobody wants to do, and everyone
| is miserable about it. Features stopped getting released,
| people leave the company, the weekly status meeting never
| makes the project seem anywhere near completion.
|
| Say "switch databases on a successful project" and I bet
| people's minds go there. That is indeed expensive and
| painful, but if you have a small team, it's just being
| agile.
|
| There is also the "punching up" / sour grapes aspect that
| can't be ignored. "I would have picked the right database
| from the start, but nobody's funding my crazy idea." Life
| just isn't fair, and HN is where you can vent, I guess.
| nrr wrote:
| I have the odd feeling that this kind of overestimation
| comes from the experience that folks tend to have in
| writing more traditional N-tier applications, especially
| post-Rails. A certain kind of database-first, ORM-first
| design philosophy really took hold after that, and it
| became somewhat common to see incredulity in response to
| telling someone that you weren't using something like
| MySQL or Postgres in production.
|
| The kinds of projects that I more generally tend to see
| make these migrations successfully are the ones that
| design with in-memory data structures first and then sort
| out how best to persist that kind of thing to disk after
| their domain model has solidified a little bit.
|
| Tailscale smells a lot like that to me.
| devmunchies wrote:
| When I used dynamic languages (Node.js and Ruby) I leaned
| on the database as the structural source of truth for
| types--it was the rigor a successful project needs.
|
| But as I've been using only statically typed languages
| the last few years (on the backend), I've moved to
| designing the domains in the code, like you said.
| powera wrote:
| Yup.
|
| When Tailscale migrates databases, it is a few engineers
| working for a few weeks solving problems that already
| exist.
|
| When a company with 50k employees does it, it takes two
| years and only makes things worse.
| norom wrote:
| I for one am enjoying this. Sure they could probably just use a
| regular ol' database, but where is the excitement in that!
| mekster wrote:
| Excitement is only for people who are learning.
| billywhizz wrote:
| you should always be learning. and excited when you are. do
| you know everything?
| mekster wrote:
| There are projects you can try new things with and projects
| that should just work when it's expected for many people to
| use. Don't push your excitement to the users.
| jitl wrote:
| SQLite is the most deployed database in the world
| isbvhodnvemrwvn wrote:
| Not for this kind of applications. Same as Java being on 13
| billion of devices, it doesn't mean anything if it's some
| config store underneath some mobile or desktop app.
| yawaramin wrote:
| > some config store underneath some mobile or desktop app
|
| Do you understand what kind of application Tailscale is?
| For all intents and purposes it's exactly what you
| described above.
| jjeaff wrote:
| I really hope they aren't taking the same tack when dealing
| with the encryption portion of their service.
| acrispino wrote:
| They're using WireGuard for the tunneling. I don't think a
| lot of people will object
| Aeolun wrote:
| What would be the analogue?
|
| Using openssl with a wrapper service? Doesn't seem
| particularly dubious to me.
| ignoramous wrote:
| > _Using openssl with a wrapper service?_
|
| Coincidentally, one of tailscale's founders already built
| this: https://github.com/sshuttle/sshuttle
| mrkurt wrote:
| sqlite is, like, the regular-est of the databases.
| isbvhodnvemrwvn wrote:
| Not quite, I got bitten by weak type system more than once.
| dangoor wrote:
| Enough people have had this complaint that sqlite has added
| STRICT tables: https://www.sqlite.org/stricttables.html
| isbvhodnvemrwvn wrote:
| Oh, finally! I haven't heard about it. Thank you!
| mekster wrote:
| Funny in here, people praise SQLite but dismiss MySQL when
| it's tiny bit off the SQL standard and people complain how
| PHP converts values with best effort guess on different
| types but not when SQLite does that too.
|
| Don't know why anyone would use SQLite in production except
| for as some KV store embedded in an app.
| yawaramin wrote:
| Adding to what tptacek said, it seems you're also out of
| date with SQLite, they added strict typing recently.
| tptacek wrote:
| Because an SQL engine that defers control over
| connections, concurrency, storage, and process management
| to users is extremely valuable for multiple reasons,
| including simple cases like caches where you'd ordinarily
| just store things in flat files or BerkleyDB-style
| database, and complex cases like distributed statekeeping
| where existing SQL databases aren't going to do anything
| close to the right thing for your application, but you
| also don't want to reinvent SQL and query planning and
| write-ahead-logs to build your complex database.
|
| The vibe I get from this thread is that a lot of people
| have only experienced SQLite from the vantage point of
| something like Rails running in a "test" environment, and
| are used to explaining to people on some PHPbb somewhere
| that you can't really use SQLite to back a multiuser SAAS
| app. That's as may be, but when you're building infra
| components, you control who the "users" are; not
| everything is a web server taking requests from all
| comers.
| nooorofe wrote:
| there is a strict mode
| https://sqlite.org/src/wiki?name=StrictMode
| coderdd wrote:
| I was pondering about using simple decentralized databases that
| are kept synchronized asynchronously.
|
| I arrived to a design where SQLite databases would be
| synchronized by Kafka. Kafka is really robust, and has a friendly
| semantics when configured to be in-order delivery.
|
| The catch is, you don't issue writes to SQLite anymore. You write
| Kafka messages, and have to prepare to resolve concurrency
| problems at read+write-to-db time. For example, user registration
| is not an atomic action anymore - you write a registration
| attempt message, and then it might race with a concurrent
| registration attempt from an other node (say due to retries and
| random load balancing).
|
| You resolve the race in message-reception time (doable due to
| stable order on multiple reader nodes). I expect this would work
| nice, but needs you to rethink all actions that used to be
| synchronous db writes.
| lifty wrote:
| There are more robust solutions out there for this
| architectural pattern. Look into dolt(dolthub) and noms.
| stavros wrote:
| I love this. If it works, keep it until shortly before it stops
| working. The older I get the more love I have for simple
| solutions that are easy to work with (and change when you've
| grown out of them).
|
| These days I wince when I hear of overcomplicated solutions for
| an MVP that "will scale" at a nebulous point in the future, while
| we're going to be paying the complexity penalty every day for a
| benefit that may never come.
| bob1029 wrote:
| This is great. We've been using sqlite in production as our
| exclusive database storage solution for over half a decade. For
| non-trivial, multi-user systems too.
|
| Replication of data has always been a concern for us, with the
| current mitigation being periodic VM snapshots. In order to
| engage larger customers, we would need to tighten this up.
|
| Our next gen data storage technology leverages some dark magic
| from fintech. Append-only log, single writer, synchronous
| replication, etc. Most recent benchmarks _with_ replication to 1
| sync witness on same lan are in excess of 200k business objects
| created or updated per second.
| nosefrog wrote:
| There's a lot of negativity here, so I just wanted to comment
| that I think the approach Tailscale is taking with sqlite is
| pretty neat. MySQL and Postgres are both pretty complicated, so
| gluing together two simpler systems, sqlite and litestream,
| sounds appealing, and I'm interested to hear how it turns out in
| the long run.
| bestinterest wrote:
| Agreed, it's a bit of a shame for HN imo. I see a team who is
| being brilliant and understanding their domain down to the core
| with as minimal resources as they can get away with.
|
| There is incredible value in incrementally adding complexity
| only when needed.
| billywhizz wrote:
| tailscale = YAGNI to the MAX! i applaud their willingness to
| think outside the box 99% of devs seem to be happy to squeeze
| themselves into just because they read about it in a book by
| some self-appointed tech god.
| tptacek wrote:
| Threads on HN are very sensitive to initial conditions. A very
| negative (and I think pretty facile) comment took an early lead
| here, most likely because of its snark/dunk value. Give the
| thread time, it'll even out.
| hokus_p2 wrote:
| [deleted]
| [deleted]
| acrispino wrote:
| Been reading HN a while and it's disappointing to see this much
| unjustified negativity.
| petethepig wrote:
| I set up a similar system in the past with redis replicas -- it
| was about 200 MBs of data that I needed to have on quick access
| to on ~ 300 servers around the world -- it was easy to set up and
| worked really well.
| [deleted]
| throwdbaaway wrote:
| It has been almost 15 months since the previous blog post:
|
| > Unfortunately, if we want to move to running more than one
| CONTROL process simultaneously for high-availability and better
| release management, that means we no longer have exactly one
| process managing the data
|
| Do you guys still only have 1 CONTROL process?
| jmconfuzeus wrote:
| That was a fun read but I'm missing something here. I've never
| worked on large scale systems and I don't understand why they
| didn't choose My or Postgre SQL.
|
| Some comments say that it's because they're complicated but what
| exactly is complicated about them?
|
| Can someone with more experience explain to me what would have
| happened if, for example, they used PostgreSQL from the
| beginning?
|
| P.S Apologies if this is a stupid question.
|
| Edit: Looks like I hurt someone's feelings in this comment.
| hokus_p2 wrote:
| billywhizz wrote:
| i don't think it's a stupid question and not sure why you
| getting downvoted. the earlier blog post from tailscale goes
| into more detail about how the system in question works:
| https://tailscale.com/blog/an-unlikely-database-migration/
|
| > Tailscale's coordination server, our "control plane", has
| become known as CONTROL. It's currently a single Go process on
| a single VM.
|
| if they had used postgres/mysql or some other external RDBMS
| then that would have meant managing a whole other set of
| dependencies _and_ doing _at least_ an IPC /Pipe roundtrip for
| every single query instead of being able to use an embedded
| database like custom-built JSON db or SQLite which is only
| going to need in-process syscalls for reading/writing to
| disk/cache.
|
| so, this allows them to scale their queries vertically orders
| of magnitude more than an external db (SQLite can do an insane
| amount of simple read queries per second) as well as making
| their whole dev/test/release cycle simpler and easier to
| manage.
|
| that's my understanding anyway - hopefully it makes some sense.
| tartakovsky wrote:
| April fool's?
| AtNightWeCode wrote:
| April fools joke? I do not get it.
| amithgeorge wrote:
| Maybe I missed something. How do you account for the VM downtime?
| The VM containing the service writing to the sqlite db, it can go
| down without warning. How do you mitigate that?
| nikita2206 wrote:
| It doesn't. Given that it periodically dumps the WAL to an S3
| bucket for actual storage this sounds like async replication.
| Async replication is prone to lose last bits (last by time) of
| data when the node goes down. They're probably more or less
| fine if the SQLite file is stored on the EBS though, it means
| they'll just have to restart that node that failed or reattach
| EBS volume to a new healthy node. It does mean downtime though.
|
| On another note I'm using Tailscale and it's a wonderful
| service and I get it for free.
| AtlasBarfed wrote:
| ... this is just replication of a database? As in the full
| database per node/replica?
|
| Hasn't AWS's SQL db-as-a-services had this for years now?
| butlerm wrote:
| Pretty sure you could track down instances of this technology
| going back decades now. Nothing is new under the sun. Doesn't
| mean you can use it.
| detaro wrote:
| I don't think they are claiming to have invented database
| replication, so what?
| ignoramous wrote:
| Tailscale mention they want to run tests locally, not deal with
| MySQL/Postgres ops overhead themselves, and also avoid vendor
| lock-in (rules out RDS / Aurora, PlanetScale, CockroachDB,
| YugaByte, Spanner et al).
|
| May be Oracle remains an option. ;)
| pid-1 wrote:
| Maybe I'm doing something wrong, but our team has like a few
| dozens of PostgreSQL RDS running on AWS for the past four
| years and the ops is... Basically zero?
| jlokier wrote:
| I don't see how CockroachDB ends up in your list of DBs that
| lead to vendor lock-in or being unable to run a cluster
| locally, because it's open source and you can run your own
| cluster. Is there something about it I haven't spotted that
| puts it in that list?
| ignoramous wrote:
| Ops overhead would be my guess.
|
| Take a look at https://litestream.io/guides/systemd/ and
| compare that with CockroachDB's.
|
| In all fairness, litestream is no
| CockroachDB/PlanetScale/YugaByte; but why pay the ops
| overhead (and your lawyers to make sense of their "OSS"
| licenses) when litestream would do just as nicely.
|
| TFA goes (emphasis mine):
|
| _Litestream..grabs a lock so that no other process can
| checkpoint. It then watches the WAL file and streams the
| appended blocks up to S3, periodically checkpointing the
| database for you when it has the necessary segments
| uploaded. ___This gives you near real time backups and lets
| you replay your database from S3 trivially..It 's a great
| hack, and a small enough program that I could read my way
| through it entirely before committing to it.____
| avinassh wrote:
| CockroachDB is licensed under BSL, so it is not really open
| source rather a source open or source available project. I
| checked their LICENSE file [0] and it mentions bunch of
| other open source licenses with BSL. I randomly checked a
| code file [1] and it seems the code is licensed under BSL.
|
| [0] - https://github.com/cockroachdb/cockroach/blob/2c4e2c6
| /LICENS...
|
| [1] - https://github.com/cockroachdb/cockroach/blob/2c4e2c6
| /pkg/kv...
| IshKebab wrote:
| Very interesting. I'd never heard of it but apparently
| MariaDB uses it too.
|
| TL;DR is the source is available but you can't use it _in
| production_ without paying a fee. However versions older
| than 4 years become truly open source, so you can use old
| versions for free.
|
| Interesting compromise. Might be better than "open core".
| vvern wrote:
| That's not quite right. You can very much use the bsl
| code in production without a license and many, many
| companies do. The BSL contains a space for additional use
| grants. In crdb's license, you can use the product for
| anything except building a hosted database as a service
| product. The definition of what that is is a little bit
| vague but it more or less means that the users of the
| product you sell shouldn't be able to cause schema
| changes. This is all just protection against cloud
| providers taking the product and reselling it. I believe
| also the terms of this bsl are that it becomes Apache 3
| years after the code has been released.
| jeffffff wrote:
| they've already had to migrate twice due to questionable
| technology decisions. having to migrate again due to needing
| to change cloud providers seems a lot less likely than having
| to migrate again because their outside the box technology
| choice didn't pan out.
| rablackburn wrote:
| I loved this article, despite the negativity.
|
| A discussion that keeps recurring on HN is how you can get a lot
| done with simple, boring solutions. This is a perfect example of
| that in practice. What's the simplest thing that'll work? Do
| that, monitor the solution, and when you start hitting a
| limitation reevaluate what the next appropriate solution is.
|
| There are plenty of comments here deriding this as insanity when
| we have established best practices, but "best practices for me
| are not necessarily best practices thee".
|
| People aren't perfect, not every engineer comes fully-formed
| grokking the industry's best solutions and their trade offs.
|
| This is what learning looks like. This is what an evolving system
| looks like. And this is how valuable software gets built every
| day.
|
| Who gives a fuck about scalability on a poc? "Let's just put in a
| json file". I Love it. Personally I start at the "use SQLite"
| step and go from there because SQLite is such amazing software.
| Thanks to this post I'm excited to try out Lifestream.
|
| You might rightly point out that Tailscale isn't a side-
| project/poc anymore. But that's exactly why they're changing to
| SQLite and writing this post; this is their team learning to deal
| with larger problems and changing needs, and they're sharing that
| journey with us.
|
| As long as they're not _ignoring_ serious risks let them
| experiment. But try not to get too upset if they're making
| different risk/effort trade-offs than you would. Even if it's
| "objectively" the "wrong" thing.
|
| My rubric for technical sanity is:
|
| - are you taking (and testing) backups?
|
| - is your infra/data secure?
|
| - are you monitoring what's happening?
|
| - can you recover from catastrophic failure in a acceptable time
| frame?
|
| - are you meeting your legal obligations?
|
| Cover those needs and you can be forgiven a lot
| suboptimal/experimental implementation details.
| mettamage wrote:
| I favorited your post. It has taken out some insecurities that
| I had about best practices. I couldn't explicitly formulate
| them either. Thanks for the (professional/career)
| advice/therapy :)
| [deleted]
| ramraj07 wrote:
| The question is, is this the simple solution? Perhaps I and
| others have been spoiled by cloud offerings but having set up
| RDS once and never having to worry much about the db
| architecture after that for years as we scale has maybe spoiled
| us.
| rablackburn wrote:
| I'll be honest and admit my own bias to cheaper, self-managed
| tooling. I'll admit that RDS definitely meets the criteria of
| simple.
|
| From what I understand RDS works great and you're happy to
| pay what feels like the tiny cost...right up until it doesn't
| anymore.
|
| You're paying Amazon a premium for them to manage the tedious
| stuff. But sometimes dealing with the tedious is what teaches
| you how the stack works and makes you jmuch more prepared to
| solve those more complex problems down the road.
|
| The uncomfortable fact is businesses finding themselves in
| situations where they experience vendor-lock-in just because
| they've let their capability wither away.
|
| Clearly David's point about avoiding MySQL and PostgreSQL due
| to fears of "vendor lock-in" resonates with me, even if I
| personally think that's not quite where the threat is.
|
| ...for Postgres anyway. I've heard enough stories that I
| never want to touch an oracle product in my career.
| ignoramous wrote:
| > _People aren't perfect, not every engineer comes fully-formed
| grokking the industry's best solutions and their trade offs._
|
| The thing is, David Crawshaw is being extremely honest and
| humble here in admitting their shortcomings and how they're
| learning from it... but an entire troupe of Silicon Valley engs
| seem to have lost sleep over it, for reasons beyond me.
| tptacek wrote:
| Are these even shortcomings? They made JSON scale for 18
| months, which is an achievement. They seamlessly switched to
| etcd, which is one of the "right things" for distributed
| state problems. Like us, they discovered that when you adopt
| a "distributed state" solution, you inherit all the API
| limitations those things come with because nothing really
| solves (or can solve) the "multiple writers multiple readers
| real-time consistency" problem. Nothing broke, but etcd was a
| pain to work with: anything you store in etcd, the developers
| that work with it have to understand the rituals of etcd.
| Wouldn't it be nice to have that just be a SQL database? Oh,
| look: that's easy to do: use sqlite, and ship WAL segments.
|
| There is negativity on this thread, but it strikes me as
| incredibly, even embarrassingly, ill-informed.
| dang wrote:
| Related:
|
| _Why I Built Litestream_ -
| https://news.ycombinator.com/item?id=26103776 - Feb 2021 (176
| comments)
| timcavel wrote:
| infogulch wrote:
| They're using litestream [1] to replicate a SQLite database,
| which streams additions to the SQLite WAL file to object storage
| and can replay it back. This is fairly hands-off from SQLite's
| perspective. There's also the SQLite Session Extension [2] that
| is a built-in way to support generating and applying "patches".
|
| I'm curious how these tools will mature, it seems like a good
| match for microservices.
|
| [1]: https://github.com/benbjohnson/litestream
|
| [2]: https://www.sqlite.org/sessionintro.html
| silisili wrote:
| I watched an interview of Mr. Hipp, creator of SQLite, that I
| can't find now but was pretty interesting. Aside from being way
| different than I expected for some reason, in a good way - very
| down to earth and friendly, he was asked specifically about
| that, and more or less answered that his job was to write a
| solid DB, and replication can be done elsewhere. That's a
| pretty honest answer, and looks like someone took up the
| challenge.
| billywhizz wrote:
| probably this one? https://corecursive.com/066-sqlite-with-
| richard-hipp/
| qbasic_forever wrote:
| He was on an episode of the Changelog podcast in the fall and
| went into similar topics:
| https://changelog.com/news/RRAw/visit It's a good listen.
| Xeoncross wrote:
| The litestream project was created by
| https://github.com/benbjohnson who wrote
| https://github.com/boltdb/bolt (a key value store) which has
| been instrumental (from my point of view) in the Go community
| as one of the original choices for an embedded database as it
| was not only fast, but had transactions with stable snapshots.
|
| It was used by https://github.com/blevesearch/bleve,
| https://github.com/etcd-io/etcd, and number of other projects.
|
| These days, https://github.com/dgraph-io/badger is often
| favored because of it's improved write throughput.
| otterley wrote:
| It should be noted that BoltDB has been retired as it
| suffered from a number of congenital defects. It was
| implicated in the major Roblox outage that happened a couple
| months ago, and Consul doesn't use it anymore.
|
| https://news.ycombinator.com/item?id=30015913
|
| (This comment is _not_ to suggest that the Litestream project
| is of poor quality or anything like that.)
| benbjohnson wrote:
| BoltDB author here. Just to clarify, the project was
| retired because of maintenance burden. CoreOS wanted to
| make some changes but I didn't have the bandwidth to test
| and maintain it all. Since this was before Go had good
| version management, we decided that they could fork the
| project as "bbolt" and users could move over as needed.
| They did a good job maintaining it so I eventually archived
| the original project.
| infogulch wrote:
| Another one of Ben's projects is on the front page right now:
|
| Postgres wire compatible SQLite proxy -
| https://news.ycombinator.com/item?id=30875837
| benbjohnson wrote:
| Bolt author here. Badger is a good database but it's mostly a
| tradeoff of write versus read throughput, specifically, range
| queries. Badger is an LSM so it doesn't perform as well (last
| time I checked) with iterating over ordered key/value pairs.
| LSMs have bloom filters to speed up point queries so those
| aren't as much of an issue.
| tapirl wrote:
| It is strange that, although
| https://github.com/syndtr/goleveldb is used in the go-
| ethereum project for years, it is mentioned rarely (comparing
| to other go dbs).
| chubot wrote:
| Honest question: what do the sqlite authors think of litestream?
|
| I seem to recall Richard Hipp on the "changelog" podcast
| mentioning it, but I don't remember what he said.
|
| 2 episodes here: https://changelog.com/person/drh/podcasts#feed
|
| I think it was very neutral, something to the effect of "there
| are multiple solutions".
|
| But I know essentially nothing about sqlite internals so I can't
| judge, or maybe that's why I didn't understand what he said.
|
| I guess I'm wondering if it's a recommended/supported mode of
| operation in sqlite. What are the failure cases? How much data
| can you lose?
| benbjohnson wrote:
| > Honest question: what do the sqlite authors think of
| litestream?
|
| Litestream author here. Dr Hipp and his team reached out when I
| first released Litestream and we had a video call together.
| They were fantastic. Really friendly and down to earth. I
| explained how I put together Litestream and we went back and
| forth on different approaches. They were really helpful with
| understanding some of the shared memory stuff that's what makes
| the new read replication work. They ended the call by asking me
| if there's anything they could help with or anything I needed.
|
| > I guess I'm wondering if it's a recommended/supported mode of
| operation in sqlite.
|
| I don't think the SQLite team endorses any tooling outside of
| what they build AFAIK. So in that sense, no, it's not
| officially supported. However, the API for maintaining
| checkpointing is publicly available and there are docs for
| controlling it from outside processes (which is what Litestream
| does).
|
| > What are the failure cases? How much data can you lose?
|
| Litestream is designed so that it keeps retrying in the event
| that you can't connect to a replication destination. S3 is
| pretty reliable so it may be a network outage that could cut
| you off. You can also enable Prometheus metrics to be reported
| out of Litestream if you want to add monitoring and alerting.
|
| As far as data loss, by default it's setup to bundle database
| changes together every 1 second, compress them, and upload them
| to somewhere like S3. So your window for data loss is 1 second
| unless S3 goes down.
|
| You can also run regular backups with the SQLite CLI and upload
| those as a fallback. That's what I typically do since it's
| really cheap and easy to setup an hourly cronjob and I'm overly
| paranoid. :)
| infogulch wrote:
| Wow that's great that you were able to discuss with the
| SQLite team directly. Did you discuss how litestream compares
| to the session extension?
| benbjohnson wrote:
| We didn't discuss the session extension. That extension
| solves a different problem of having multiple primaries
| that need to merge their data. I haven't used it personally
| but it looks great for that purpose.
| chubot wrote:
| Cool, thanks for the response!
| mathgladiator wrote:
| I believe using a large JSON file is not half bad, but you do run
| into the problem of how do you index and query it in meaningful.
| I actually ran into this problem when building a game because a
| document doesn't provide a great model..
|
| The relational model is... JUST... SO... GOOD. And, it is a shame
| that most of the relational systems are so complicated.
|
| A document within Adama (https://www.adama-platform.com/) is
| basically a giant JSON file held within memory with clients
| connected via a WebSocket. I'm basically building my own indexing
| since I want the indexing to be reactive, and I've got
| exceptionally fun optimization problems.
|
| Litestream, in my opinion, is a great way to get started.
| Actually, it's beyond fantastic if you maintain the 1 server to 1
| database because migrations become so easy. So, I applaud the
| team for taking this simple approach.
| tptacek wrote:
| If you have a "single source of truth", sqlite/Litestream seems
| like a pretty-near-optimal way of taking advantage of the
| relational model while keeping design simplicity. I don't know
| what the higher-level architecture of Tailscale is, but we have
| the same problems; we have a complicated "single source of
| truth" that takes the form of a Consul cluster, but sqlite
| makes an absolute ton of sense for us, because we can condense
| the Consul cluster down to a SQL schema and then ship it around
| our fleet.
|
| (we don't use Litestream right now; we're pushing the
| complexity up a layer in our design instead)
|
| A lot of people on this thread are looking down their noses at
| sqlite, but I think they're kind of beclowning themselves; the
| unreasonable effectiveness of sqlite has been a meme in infra
| dev for a couple years now. It's not a new idea. Lots of people
| are doing stuff like this.
|
| The funniest bit on this thread is the person saying they
| should use RDS, as if their infrastructure was just a big Rails
| app.
| statictype wrote:
| It would be interesting to known why a standard boring RDS
| setup wouldn't solve their problem completely. In fact I
| would be more interested to understand that than the actual
| details of sqlite tailing.
|
| (I think the reason they gave was vendor lock-in, but apart
| from that, I didn't understand why it wouldn't be adequate)
| tptacek wrote:
| One obvious reason not to use RDS is wanting disk-local
| caches of information replicated from a single leader,
| rather than having every single machine in your fleet
| calling out to an external service _on every read_. That 's
| certainly why we're not considering Postgres in our
| infrastructure, even though managed Postgres is a product
| we in fact offer.
|
| We use Postgres! It's the backing state for our API, and an
| important source of truth in our architecture. Postgres is
| a great way of serving a GraphQL API. It is not necessarily
| a good way to back an infrastructure service.
| statictype wrote:
| Makes sense once you think of Tailscale as an
| infrastructure-level service and not just an app. Thanks
| throwdbaaway wrote:
| Sounds like postgres listen/notify could be viable for
| your high-read-low-write use case? Or it is not scalable
| enough for the fleet size?
| tptacek wrote:
| That's still all your infrastructure components calling
| out to an external service on every read --- and for what
| advantage?
|
| This isn't an app server; it's an infrastructure
| component, running (I don't know about Tailscale here,
| but we use SQLite in similar uses cases) on potentially
| hundreds or thousands of machines.
| broken8ball wrote:
| Really curious...how do you meaningfully overlay any indexes on
| top of a big-ass JSON file? Technical details are appreciated,
| and no problem if it's your secret sauce--just very curious how
| this is accomplished!
| acidbaseextract wrote:
| The normal way? You can implement whatever kind of index you
| like -- b-tree index, bitmap index, hash index are all useful
| and conceptually simple if you're familiar with the backing
| data structures.
|
| For example, if you want to index a "foreign key" id stored
| in each "record" in a JSON array of objects, you build a hash
| table from the FK id values to the JSON array indices of the
| objects that have that id. It can be as stupid simple as an
| `fk_index = defaultdict(set)` somewhere in your program, to
| use a Pythonism.
|
| Now when someone wants JSON objects in that array matching a
| given FK id, they can just O(1) look in the index to know the
| position of records that match. Much better than an O(N) scan
| of every item in the array.
|
| Of course you have to to maintain the index as writes to the
| JSON happen, but that's not bad once you understand how
| things work. No real secret sauce.
| mathgladiator wrote:
| The secret sauce may be the need to take control of the
| write path.
| mathgladiator wrote:
| So, a funny thought experiment is what happens when you parse
| a JSON file at the same time? You also index by primary key
| (the field name).
|
| So, I mirror this thinking and having be just an object with
| the keys being the primary key. Then, I simply index all the
| children by their fields based on insights from the developer
| via the index keyword.
|
| So, if you have
|
| record R { public int id; client int owner; int age; index
| age; } table<R> rows;
|
| then queries for age can be accelerated by the table.
|
| like "iterate rows where age==42" will basically hone in on
| the bucket of age==42. I currently only index clients by hash
| and integers.
|
| The critical aspect which makes this work is that I monitor
| all mutations. When a child object has a field mutated, then
| it is removed from all indices and placed into an unknown
| index. Any queries will also consider it as the purpose of
| queries to simply narrow the field. Once data changes are
| persisted, the index is updated and items are moved out of
| the unknown bucket. This works fairly well because the
| indices are primarily used during the privacy check phase.
| bArray wrote:
| (Ignoring the fact this may be an April fools joke.)
|
| > We're going to put everything in a single file on disk again.
|
| For most small scale projects I'm just reading and writing lines
| to a file on disk. Sure, I could over-engineer some database, or
| I could get a minimal viable project off of the ground and worry
| about scaling later. Most projects run into issues long before
| scalability.
|
| I found myself recently wanting a lightweight key-value store for
| random structures. For this purpose, I created a toy single-file
| key-value store [1] in less than 256 lines of C. It's not quite
| ready for prime time, but it's incredibly easy to use.
|
| [1] https://gitlab.com/danbarry16/u-database
| KwisaksHaderach wrote:
| _The obvious candidates were MySQL (or one of its renamed
| variants given who bought it) or PostgreSQL, but several of us on
| the team have operational experience running these databases and
| didn't enjoy the prospect of wrestling with the ops overhead of
| making live replication work and behave well. Other databases
| like CockroachDB looked very tempting, but we had zero experience
| with it. And we didn't want to lock ourselves into a cloud
| provider with a managed product like Spanner._
|
| What about managed mysql/posgresql? no lock-in and installing
| them locally is trivial.
| mrkurt wrote:
| Managed DBs have tremendous lock in. Just try migrating off RDS
| with zero downtime. You can't, because they've "managed" your
| ability to configure external replicas. Then built a whole
| brittle data migration service that probably won't work for
| your DB.
| Winsaucerer wrote:
| I'm having a really difficult time migrating my managed
| postgresql instance in Google Cloud just to get it to a newer
| version. I'm down to my last option: shut down everything,
| export, re-import somewhere else again.
|
| I did briefly have logical replication working, and so had my
| near zero downtime solution in hand. But the replication
| broke after a few days because I delayed cutting over, and
| then I couldn't get it going again. I couldn't find out how
| to debug this either.
|
| The lesson I've learned here is that I'll be avoiding managed
| DB products whenever I can. It takes away control that turns
| out I sometimes really need.
| jeffffff wrote:
| they've already had to migrate twice due to questionable
| technology decisions. having to migrate again due to needing
| to change cloud providers seems a lot less likely than having
| to migrate again because their outside the box technology
| choice didn't pan out.
| tptacek wrote:
| "Questionable technology decisions". You're trying to dunk,
| but I don't think you understand where the hoop is. Their
| technology decisions have panned out _Tailscale well_. We
| should all be so fortunate. This isn 't Twitter with the
| "fail whale"; the only reason you know about any of this
| stuff is because they wrote about it. They ran their entire
| service with a JSON file backend for 18 months, and
| switched from it to etcd without you even knowing about it.
| hokus_p2 wrote:
| The etcd solution was highly questionable. Actually, a
| prime example for reinventing the wheel, but worse.
| jeffffff wrote:
| i really don't know a ton about this product or team but
| it sounds like if they had used aurora mysql or aurora
| postgres in the first place then there would be nothing
| to write a blog post about because it would've just
| worked and kept working. they say they want to avoid
| vendor lock-in but if the vendor became a real issue
| they'd be doing their first migration instead of being on
| v3 already. additionally, their bespoke solution relies
| on s3, which is also a vendor-specific technology, so it
| seems like they haven't avoided vendor lock in? i've seen
| many cases of developers doing more work to avoid vendor
| lock-in than it would take to replatform if it ended up
| being a necessity, and this really feels like that
| looking at it from the outside. i'd understand this
| better if mysql or postgres couldn't solve their problem,
| but that is not the case here, and i can't wrap my head
| around a company who is ok with their devs reinventing a
| very good wheel 3 times when the obvious choice would've
| worked fine the first time. it seems like they are
| successful in spite of these decisions, not because of
| these decisions.
|
| https://mcfunley.com/choose-boring-technology
| hokus_p2 wrote:
| >> it seems like they are successful in spite of these
| decisions, not because of these decisions
|
| That is my conclusion, too.
| mrkurt wrote:
| The thing about startup decisions is: most of them are
| "wrong". Or they start right and become wrong later.
| Successful startups aren't successful in spite of their
| wrong decisions, they're successful because they can
| change them very quickly (then write a blog post about it
| and get more customers).
|
| There's also a strong correlation between people who look
| at things from weird angles and also build good products.
| Why are you surprised the people who invented an entirely
| new way of doing VPNs also don't cargo cult database
| storages?
| jeffffff wrote:
| it's not cargo culting if it works, and they even say in
| the article that mysql or postgres would've worked
|
| maybe next they will stop cargo culting operating systems
| and switch to SerenityOS?
| tptacek wrote:
| They're talking about running many instances of MySQL
| locally, not hooking all their systems up to RDS.
| tptacek wrote:
| You should start by learning more about the product, and
| _then_ tell them they should use Aurora for all their
| backing store.
| jeffffff wrote:
| by their own admission in the blog post mysql or postgres
| would've worked fine, but it seems that they have an
| extreme aversion to doing what everyone else does and are
| justifying it with this vacuous vendor lock in argument.
| can't wait for what "A database for 2023" looks like!
| tptacek wrote:
| You don't know what this database is --- upthread, you
| said you don't even know what the product is. All you
| appear to know is that they should be using something
| like RDS. Isn't that a weird position to take?
| jeffffff wrote:
| it's also kinda weird to write a blog post about a
| database being used for what is apparently a very
| specific use case without mentioning what that use case
| is. the burden of proof is on tailscale to explain why
| they need to deviate from industry norm here and clearly
| from the entire comment thread of people wondering the
| same thing as me they haven't done that. this blog post
| might actually be valuable if they included more context
| so that people could learn when something like this might
| be a good idea, especially since it's not a good idea
| >99% of the time. as is, no one should be surprised by
| this response.
| tptacek wrote:
| This is one of several comments you've written where
| you've acknowledged you don't know what the use case is.
| But you've stridently insisted that they should have used
| Aurora Postgres or Aurora MySQL. You get how strange this
| take is, right?
|
| One thing that would have helped your writing on this
| thread: a lot more question marks. It's OK not to
| understand something! Asking questions helps everybody.
| yawaramin wrote:
| Sounds like you need to do a lot more research before
| commenting on this? It's pretty easy to find in the
| Litestream docs that it replicates to S3, Google Cloud
| Storage, Azure, and other options even including SFTP. In
| fact by off-loading the storage integration details to
| Litestream, the Tailscale people now get seamless storage
| vendor independence almost for free.
|
| This is what a really smart and future-proof solution
| looks like.
| jeffffff wrote:
| or they could have used mysql or postgres hosted by aws,
| gcp, azure, etc? i would put money on this not being the
| last time they change databases
| tptacek wrote:
| Again: this is an infrastructure component, running on
| many machines internally. It's not a Rails app. We use
| SQLite in very similar circumstances across hundreds of
| machines. Having all those machines schlepping all their
| reads back to RDS would not only be untenably slow, but
| it would also make the whole system less stable. I don't
| think you've really thought through the design at all,
| and you should before making comments like these.
|
| I'd put money on this not being the last database change
| too! But not for the same reasons you would.
| yawaramin wrote:
| I'll take that money quite easily ;-)
| bobnamob wrote:
| The s3 api is rapidly becoming a distributed storage
| standard. Building your product around it is hardly lock-
| in these days.
|
| AWS does offer one (very reliable) implementation but
| they're very definitely not a monopoly.
| IshKebab wrote:
| They're not questionable because they didn't work,
| they're questionable because they were so difficult to
| run that they had to migrate _twice_.
|
| You could absolutely run a web site using Brainfuck
| without any failures or any customers realising it.
| Doesn't mean it's not a questionable decision!
| growse wrote:
| Choosing a technology that supports your business needs and
| growth, and that also lets you easily migrate to something
| else once it stops being suitable (or your requirements
| change, or you discover a way in which the tech doesn't
| work for you, or whatever), doesn't sound like a
| _questionable_ technology decision, it sounds like a
| _great_ one.
| butlerm wrote:
| If you install it locally, that is pretty much the definition
| of an unmanaged database. Other than by the user of course.
| alexjurkiewicz wrote:
| You can use a managed version of the database in production,
| and an "unmanaged" local Docker image during development.
| didip wrote:
| So... litestream is bidirectional? Nothing can go wrong when
| daemon A and daemon B write almost at the same time?
|
| Edit: I just found this tips and gotcha page:
| https://litestream.io/tips/
| benbjohnson wrote:
| Litestream author here. I'm glad you were able to track down
| what you were looking for. For posterity, I'll add a response
| here. Litestream is a one-way, physical replication tool so you
| have a single primary node that continuously copies changes out
| to an external destination (S3, SFTP, etc).
|
| If you're looking to have multiple primaries then you can try
| using the SQLite session extension[1] and copying patch files.
| That's quite a bit more complicated though.
|
| [1]: https://www.sqlite.org/sessionintro.html
| flatiron wrote:
| How did they think going from a single json to either MySQL or
| Postgres?
|
| I know they say this isn't April fools but I'm not sure I would
| put it past the engineering team. This can't be real.
|
| "We looked at real solutions like cockroach but nobody could be
| bothered to rtfm"
| eminence32 wrote:
| I'm sympathetic to the idea of avoiding a database because no
| one on your team has experience with it.
|
| On my old team, our product supported Oracle DB (along with
| some other databases), but on one on the team really was an
| oracle expert, and we would frequently run into questions that
| we didn't really know how to answer. We would have dropped it
| if we could.
|
| It's easy to say "rtfm" but doing this doesn't make you an
| expert in the system. And as I bet many NH readers know,
| there's a big difference between "I read the quickstart guide
| and got a DB working in an afternoon" and "We've hit an error
| an error that the manual says nothing about, but our DB expert
| saw this once, 7 years ago, and remembers how to recover"
| alophawen wrote:
| Not commenting on tailscale, but for the state of databases.
|
| Sometimes boring is the right choice. PostgreSQL has worked for
| decades now, and seems to have regained much of the performance
| that MySQL once boasted.
|
| If you do this for the money, investing in tried and true (but
| boring) software should be the default solution.
|
| I watched with dread how the MongoDB fiasco played out a decade
| ago. Meanwhile, I kept using PostgreSQL and had a good ride.
| kfk wrote:
| It depends. I was in a demo call from Blaze, a CDP (Customer
| Data Platform), Segment.io competitor. Their underlying db
| seems to be MongoDB and for a CDP makes a lot of sense. In a
| CDP you collect all kind of data on n hard ids that identify 1
| person interacting with your web assets, schemaless and json
| first is a lot easier to reason in this context than SQL. This
| is because you keep enriching your profile with additional
| attributes over time. Can PostgreSQL do this? Absolutely, but
| it's not its main feature.
|
| Things are complex, especially in data, I think it's important
| to evaluate use cases rather than going with generic
| assumptions like "use boring tech".
| louwrentius wrote:
| You say 'sometimes', I think it should be the default position.
|
| When you want to deviate from the boring software and
| technology, there must be a very sound 'business case' for it.
|
| If you are building a planetary scale business, OK, maybe that
| warrants something more fancy.
|
| But how many companies really need that? I'm not limiting
| myself to the HN unicorns, but looking at the whole market.
| alophawen wrote:
| I agree with you and I also wrote:
|
| > If you do this for the money, investing in tried and true
| (but boring) software should be the default solution.
|
| Meaning if you are just playing around, then by all means
| evaluate new and shiny stuff.
| unfocussed_mike wrote:
| Yep.
|
| IMHO MySQL becoming boring (fast, pretty reliable, non-
| compliant quirks being ironed out, decent backup tool
| availability, credible admin tools, JSON and CTE support) is
| one of the most important long term trends on the internet.
|
| Along with the parallel evolution of PHP.
| klysm wrote:
| I strongly dislike the combination of php and MySQL, from the
| (recent) experiences I've had with it. I don't see a good
| reason to pick that as a greenfield stack, but I could be
| convinced otherwise.
| unfocussed_mike wrote:
| MySQL + PHP (Laravel + Composer + Lighthouse + Spatie
| MediaLibrary) is very productive.
| pphysch wrote:
| For me it's not necessarily the tech themselves, but the
| lack of any framework to guide development for the next
| guy.
|
| All the PHP+MySQL I've worked with has been _horribly_
| bespoke and brittle.
|
| Want to significantly change a URL? Need to refactor every
| path in the .PHP file, because it's importing something
| from ../.. Or get in the spaghetti business with path
| aliasing on the webserver.
|
| Want to refactor the database schema? Difficulty:
| Impossible, because raw SQL strings are scattered
| _everywhere_ , including on other systems.
|
| Want to add/modify a new internal CRUD form? Gonna be
| several hours of work to RE what the last guy did to keep
| things consistent, while tiptoeing into the SQL to not
| break anything. With a proper framework, this is a 5 minute
| task and a few LOC into the "admin interface".
|
| It works, and you can do a lot with just a few php files
| and a basic Linux system, but there are downsides.
| unfocussed_mike wrote:
| There are several PHP frameworks worth using.
|
| Laravel is, IMO, among the best web frameworks available
| full stop. Certainly one of the most maturely documented.
|
| It addresses all the points you make.
| alophawen wrote:
| I had a 10 year career doing PHP, in the last year or so
| I also enjoyed Laravel and built 10 or so sites with it.
| It did many things right.
|
| Eventually some blade template error resulted in an
| impossible stack trace due to the insane amount of magic
| behind the scenes in Laravel, and while debugging that I
| decided to stop relying on such black magic hack of a
| framework. Around the same time Go gained in popularity
| and I have been working with Go since and are very happy
| with it.
|
| This was Laravel 5 days and I'm sure much has changed.
|
| I completely adored the laracasts made by Jeffrey Way. I
| wish every framework had a guy like that do produce
| learning materials.
| pphysch wrote:
| Yeah I would love if these places had a solid Laravel
| deployment, but they sadly don't.
|
| And if I'm gonna deploy a new framework greenfield, I'll
| just go with Django since Python has more mindshare in my
| industry.
| rsp1984 wrote:
| > I watched with dread how the MongoDB fiasco played out a
| decade ago.
|
| Could you elaborate on this? MongoDB as a company is worth
| $30B, so it looks like they did at least some things right.
| billywhizz wrote:
| they did their marketing right.
| alophawen wrote:
| There was many new node projects using MongoDB and I remember
| several failed, but details elude me.
|
| There was a Slack competitor using MongoDB that was snappy
| and js hipster trendy that ultimately failed because of
| unreliable database IIRC.
| UncleEntity wrote:
| How many crypto currency projects went down with $million+
| losses due to using mongoDB as the backend?
| vinay_ys wrote:
| Their paid managed service works well for enterprise SaaS
| companies who need < 3-4 TB of storage max ever per customer
| cluster and have enterprise scale users (<100k users).
|
| It also works well for early stage Consumer Internet startups
| that haven't yet achieved huge growth but care a lot about
| developer productivity while churning features at high
| velocity.
|
| But it gets blamed for reliability issues suffered by highly
| successful Consumer Internet companies that have achieved
| scale and have DAU, MAU in > 10M and have lot more than 4TB
| of data in a single cluster.
|
| But that's basically a good problem to have at that stage.
| Usually such a company would have many more problems -
| monolithic application with monolithic database with huge
| unmanageable schema and indexes gone wild etc. Usual solution
| at that stage would be some sort of rearchitecture towards
| Microservices with multiple specialized databases for
| different use-cases - usually cloud hosted managed databases
| with horizontal scalability dedicated for online user-path
| workloads and separate OLAP tech stack for offline
| ETL/analytics workloads.
| database_guy wrote:
| At work I have a multi-petabyte cluster in MongoDB Atlas
| with no issues. The managed service is fantastic; I'm going
| to need some citations on how it degrades after a few
| terabytes. I don't really like MongoDB's document-model-
| everywhere approach but I have to respect how well their
| managed service works.
| brycelarkin wrote:
| It looks like you guys use AWS. Why not just use RDS if you don't
| want to deal with database management?
| epolanski wrote:
| Some engineering teams seem to take overcomplication always one
| step too far. It's very hard to estimate future work and
| overconfident engineers consistently downplay the costs.
| ignoramous wrote:
| I'm not sure why you're being downvoted. You're right:
| running databases isn't for every team! Reach out for a
| managed database if you can.
|
| But tailscale isn't some random group of engs. They've
| probably got the chops to pull off literally _anything_ they
| want to. I mean TFA casually mentions online cross-database
| transfers, multiple zero-downtime schema migrations,
| inspecting litestream 's replication code for feasibility,
| deftly modifying sqlite WAL checkpoints... all in one breath.
| jeffffff wrote:
| if they're this talented, is their time really best spent
| on dba work rather than improving the product?
| ignoramous wrote:
| > _is their time really best spent on dba work..._
|
| It _seems_ to me that tailscale engs want to avoid DBA
| work _but_ also _not_ use managed offerings, and so, they
| 're comfortable paying the costs they have to (such as
| multiple migrations).
|
| > _...rather than improving the product?_
|
| Well, you'd guess they want to be able to continually
| improve their already credible product too. When TFA
| points out that zero vendor lock-in and hassle-free,
| local end-to-end tests are non-negotiable, I think it is
| for this reason.
|
| ----
|
| > _if they 're this talented, is their time really best
| spent on..._
|
| From: https://tailscale.com/blog/go-linker/
|
| "People are often surprised and sometimes horrified when
| they learn that Tailscale maintains its own fork of the
| Go toolchain. Tailscale is a small startup. Isn't that a
| horrible distraction, a flagrant burning of innovation
| tokens?"
|
| "Maybe. But the thing is, you write code with the
| engineers you have."
|
| "We had a problem: We kept crashing on iOS, and in
| addition to being awful, it was preventing us from adding
| features."
|
| "Another team might have decided to cut even more
| features on iOS to try to achieve stability, or limited
| in some way the size of the tailnet that iOS could
| interact with."
|
| "Another team might have radically redesigned the data
| structures to squeeze every last drop out of them."
|
| "Another team might have rewritten the entire thing in
| Rust or C."
|
| "Another team might have decided to accept the crashes
| and attempted to mitigate the pain by making re-
| establishment of connections faster."
|
| "Another team might have decided to just live with it and
| put their focus elsewhere."
|
| "The Tailscale team has Go expertise, spanning the
| standard library to the toolchain to the runtime to the
| ecosystem. It's an asset, and it would be foolish not to
| use it when the occasion arises. And the fun thing about
| working on low level, performance-sensitive code is that
| that occasion arises with surprising frequency."
|
| "Blog posts about how people solve their problems are fun
| and interesting, but they must always be taken with a
| healthy dose of context. There may be no other startups
| in existence for which working on the Go linker would be
| a sensible choice, but it was for us."
| jeffffff wrote:
| > When TFA points out that zero vendor lock-in and
| hassle-free, local end-to-end tests are non-negotiable, I
| think it is for this reason.
|
| if zero vendor lock-in and hassle-free, local end-to-end
| tests are non-negotiable, why are they using s3?
| migrating to another s3 compatible backend would be
| similar in effort to migrating from aurora mysql or
| postgres to another managed mysql or postgres service or
| to self-hosted mysql or postgres
| tptacek wrote:
| First, S3 and a SQL database aren't comparable. But I
| think you're bringing up S3 because they're using
| Litestream to ship WAL frames to S3. Go read the
| Litestream documentation; Litestream syncs to basically
| anything. They don't need to "migrate to another S3
| compatible backend"; they can migrate to almost anything
| that can save a file.
|
| It's a super confusing argument regardless, because the
| industry is lousy with "S3-compatible backends".
| ignoramous wrote:
| > _migrating to another s3 compatible backend would be
| similar in effort to migrating from aurora mysql or
| postgres to another managed mysql or postgres service or
| to self-hosted mysql or postgres_
|
| You may be right. I have no experience migrating
| litestream but from the docs
| (https://litestream.io/guides/) it is literally cp'ing
| files from S3 to wherever and exec'ing one of these one-
| liners (of course, the devil is in the details):
| litestream restore -o my.db s3://BUCKETNAME/PATHNAME
| litestream restore -o my.db
| abs://STORAGEACCOUNT@CONTAINERNAME/PATH litestream
| restore -o my.db gcs://BUCKET/PATH litestream
| restore -o my.db
| s3://SPACENAME.nyc3.digitaloceanspaces.com/db
| litestream restore -o my.db s3://BUCKETNAME.us-
| east-1.linodeobjects.com/db litestream restore -o
| my.db sftp://USER:PASSWORD@HOST:PORT/PATH
| benbjohnson wrote:
| Litestream author here. Yeah, you're basically right but
| it's simpler than a DB migration. No need to copy the old
| data over. You can remove the `-litestream` metadata
| directory and point it at a new replication destination
| and it'll automatically re-snapshot the database begin
| replication.
| epolanski wrote:
| Some examples: they could get rid of that pointless
| bootstrap on their website, they are shoving almost 1.5
| MBs for a _single font alone_ on their main page and
| their HTML semantics are nowhere to be found. This will
| all impact their bounce rate, accessibility and SEO.
|
| I just don't believe the tale of "such skilled
| engineering teams" which don't show that in their
| products but blogposts.
| blizz017 wrote:
| What if I told you that the product engineering team is
| almost never the same team maintaining the website; hell
| it's likely the website is contracted out and maintained
| by the marketing team.
| nooorofe wrote:
| do you have doubts those people are skilled?
| https://en.wikipedia.org/wiki/Brad_Fitzpatrick
| https://news.ycombinator.com/item?id=21727925
| eyelidlessness wrote:
| Disclaimer: I hate doing ops, but I've been in a position to
| actively hate doing it fairly regularly for several years of my
| career. My perspective is as a person who doesn't want to deal
| with any of this kind of stuff. So I've probably failed to
| acquire knowledge which would make it less frictionful for me,
| purely from lack of interest.
|
| RDS has some significant downsides which I would personally
| consider no go if I were in a position to evaluate it. The one
| which stands out as particularly painful from my past
| experience is... it's excruciatingly slow to provision or make
| configuration changes. Like _lose whole days of work to a few
| iterations of trial and error_ slow. Combined with AWS'
| sprawling and inscrutable set of authorization and
| configuration options, the weird idiosyncrasies between most of
| their offerings, and the absolutely opaque naming applied to
| most of those offerings... trying to use RDS effectively as a
| managed database service felt more _to me_ like becoming a full
| time ops professional.
| carlhjerpe wrote:
| Idk, I just use the Terraform module and set the Helm values
| on whatever should consume the db with whatever I get back
| from the module then call it a day. Then again, I'm not
| scaling big at all.
| 0xFFFE wrote:
| Totally off topic. I am not an UI/UX designer but why can't the
| page scale to the size of the screen? Sure I could ctrl/cmd++ but
| is it hard to detect the screen size? Genuine question, not
| trying to be condescending.
| [deleted]
___________________________________________________________________
(page generated 2022-04-02 23:02 UTC)