[HN Gopher] PlanetScale is now generally available
___________________________________________________________________
PlanetScale is now generally available
Author : petercooper
Score : 209 points
Date : 2021-11-16 14:23 UTC (8 hours ago)
(HTM) web link (planetscale.com)
(TXT) w3m dump (planetscale.com)
| DeathArrow wrote:
| I only see buzzwords and hype in the post. Nothing technical,
| nothing much about why would make a good business sense to use
| their offering.
| gh123man wrote:
| This seems cool. I am currently shopping around for a cheap
| managed database for an app I am launching soon (and
| bootstrapping - so cost is a concern right now). We are starting
| small but want room to grow. The free tier looks really
| promising! Especially compared to some of the entry level plans
| on other cloud providers.
|
| Can anyone who has tried Planetscale and compared it to other
| managed database solutions tell me about their experience?
| remorses wrote:
| It is pretty awesome
|
| The dashboard is clean and easy to navigate
|
| The branching feature is really good idea, it allows you to
| confidently run migrations with no downtime
|
| The daily backups are also great, no additional steps to set
| them up
|
| They also have a graph to check query latency which is nice
|
| The only downside is that they dont have foreign keys but I
| used prisma referential integrity for that so not really an
| issue for me
| porker wrote:
| > The branching feature is really good idea, it allows you to
| confidently run migrations with no downtime
|
| So this branching feature is one I still don't understand
| properly.
|
| Let's imagine my startup is big enough to warrant a
| PlanetScale database. Hundreds of rows are written every
| second. When I branch, are those writes applied to the
| original and the branch? If so, what happens with migrations
| that mean the incoming writes no longer work? Or is the
| "confidence" it gives not about full migrations but about
| knowing that if your migration fails the original DB is still
| working?
| dewey wrote:
| I think going for a speciality database at this stage is
| premature optimization. Go with the safe choice (Existing
| tooling, knowledge, free tiers) and use Postgres or something
| else established. Later on you can always move workloads to
| other databases if needed.
| boiler_up800 wrote:
| I've tried Aurora and Firebase. I think PlanetScale wins on 2
| areas at 2 different scales: branching when you're small, and
| connections when you're large.
|
| With branching it's a feature where every dev can run a command
| and get their own database version to mess with. They can
| submit a pull request and as the tech lead I can check it and
| publish it to prod. Or integrate it with our CI.
|
| On connections, it is interesting how serverless architectures
| totally flip the model. My node servers used to make a few
| connections to Postgres while serving thousands of users. Now,
| each NextJS instance makes its own connection and I might have
| thousands of database connections for thousands of users.
| PlanetScale is the only one I know of with effectively no limit
| on the number of connections. I think Aurora tops out at
| 16,000.
| sreeramb93 wrote:
| I will bet my money that planetscale sales will not eclipse
| Amazon RDS.
| _ben_ wrote:
| Related to some of the comments here, we are building
| PolyScale.ai https://www.polyscale.ai/ which is a global database
| cache. This solves regional latency and read query performance
| (concurrency), whilst maintaining the transactionality of the
| origin database.
| pepemon wrote:
| Vitess is awesome. Nice to see that there is the managed offering
| available too now!
| candiddevmike wrote:
| Open source, serverless, developer experience, really going for
| SEO with this blog. But it doesn't say anything about what the
| product does or that it's MySQL compatible. Don't waste your time
| reading it.
| dclusin wrote:
| Also mentions the incredible journey they are on... Reminds me
| of https://ourincrediblejourney.tumblr.com/
| jfbaro wrote:
| Congrats! Amazing offering. Is there any plan to support
| PostgreSQL as well?
| notum wrote:
| It's nice that they have solved backups in a very respectable
| manner which alleviates the feeling of being locked in.
| fabian2k wrote:
| I still don't understand how you can work safely with this kind
| of pricing model. You pay $1.50 per 10 million rows _inspected_
| by the database.
|
| A relational database like MySQL has a query planner, so you're
| not fully in control over how exactly the database accesses your
| data. How many rows are read not only depends on your specific
| SQL query but also on the contents of the tables you read and the
| parameters supplied along with the query.
|
| A statistics update that triggers a bad query plan on a table
| with 10 million rows could cause a full table scan instead of
| retrieving a few rows using an index scan. Suddenly your query is
| a million times more expensive at $1.50 per query. Depending on
| how quickly and widely this scales, that could get very expensive
| very fast.
|
| 10 million rows is also not a lot for a relational database. I
| know my comparison now isn't fair as you get a lot more with the
| hosted version, but a quick and dirty experiment with Postgres
| and a small table of two ints and a text column with 10 million
| rows took ~1.5 seconds to query on my desktop, using only 1 CPU
| core. Translated into this pricing model that's something like
| $60 per minute.
| [deleted]
| danielrhodes wrote:
| This is sort of wait nails people with DynamoDB pricing. If you
| mess up your query, you end up scanning through tons of data,
| racking up the cost.
| dastbe wrote:
| imo at least with dynamodb, its pretty straightforward what
| any given call is capable of doing, and you can get back
| capacity usage on every request to track. with these
| mysql/postgres compatible apis the actual query is abstracted
| as is its cost.
| rustc wrote:
| Does MySQL (or PlanetScale) provide a way to input a query and
| get a lower/upper bound on the number of rows the query will
| "inspect" when run?
| vishnugupta wrote:
| MySQL's "explain <query>" does exactly what you described and
| better. It shows indexes used, exact number of rows scanned
| etc. It's the first thing I do when I suspect a query taking
| time. It's a pity that not too many backend engineers know
| about it now a days.
| rustc wrote:
| > exact number of rows scanned etc
|
| I'm reading MySQL docs [1] and I can't figure out which
| value in the example output corresponds to the "exact
| number of rows scanned".
|
| > It's a pity that not too many backend engineers know
| about it now a days.
|
| I use EXPLAIN with PostgreSQL when needed but mostly focus
| on the execution time, not the number of rows scanned
| (reading a large row could be orders of magnitude slower
| than reading a row in a JOIN table of two integers in a row
| based storage engine).
|
| [1] https://dev.mysql.com/doc/refman/8.0/en/explain.html
| vishnugupta wrote:
| Let's just say that MySQL docs are not _super_ helpful
| when you want to quickly refer to something.
|
| Here's[1] an explain output that I ran just now. The row
| labeled "rows" is the actual number of rows scanned which
| as you can guess is bad because it's doing a full able
| scan.
|
| I completely ignore the actual execution time and focus
| mostly on two things 1) indexes used; 2) number of rows
| scanned. If rows scanned is high it means I need to do
| something about it now even if the absolute query time is
| low. Either an index is missing or someone in the
| application code is querying the wrong way (random
| example, using a "like" on a full table).
|
| [1] https://imgur.com/a/0Pj52QA
| fabian2k wrote:
| That only explains the particular query plan chosen for the
| query at that time. This might vary depending on the table
| statistics, query parameter or moon phase if you're
| unlucky. It is a very important tool, but doesn't solve the
| problem here that in the end the database is in control and
| decides how many rows to read, not you.
| redis_mlc wrote:
| > exact number of rows scanned etc.
|
| Never heard the word "exact" used for number of rows from
| EXPLAIN.
|
| Source: MySQL DBA.
| CaveTech wrote:
| Yes, through `EXPLAIN`.
| astockwell wrote:
| Their FAQ [1] clarifies this somewhat:
|
| > Every time a query retrieves a row from the database, it is
| counted as a row read. Every time a row is written to the
| database, it is counted as a row written.
|
| [1] https://planetscale.com/pricing (FAQ at bottom of the page)
| fabian2k wrote:
| > Reads: Retrieving or inspecting rows during a query or
| mutation of any kind to your PlanetScale databases
|
| That's what the detailed pricing page says, which is somewhat
| different. The pricing is actually per row inspected, not per
| row returned to the client. So in aggregations you'd pay for
| every single row you aggregate. So I'd argue that the FAQ is
| actually misleading.
| astockwell wrote:
| Oof. Well that's not good.
| spookthesunset wrote:
| So does that make an accidental full table scan an
| expensive operation from a billing standpoint?
|
| Like others say, the end user doesn't have a lot of control
| over how the query planner does its magic. They do have
| control over what goes into and out of the system though.
|
| That being said things like Google BigQuery bill by compute
| time and data examined so I guess that works as long as the
| price per "unit" is on the order of millicents or
| something.
| mooreds wrote:
| See also https://briananglin.me/posts/spending-5k-to-learn-how-
| databa...
| nojito wrote:
| >What's more? This allows you to pay only for the database
| resources you use, on a second-by-second basis. ..."
|
| This is the biggest sham in technology offerings today. The idea
| that you only pay what you use is a misnomer.
| dmarlow wrote:
| How so?
| ec109685 wrote:
| Right, should be resources you "reserve".
| ComputerGuru wrote:
| So does this share MySQL's weaknesses? As someone that used MySQL
| for over a decade before seeing the light and switching to
| Postgres, I'd be more reassured if this wasn't compatible with
| any existing RDBMS than with it being MySQL compatible.
| gmiller123456 wrote:
| What made you switch to Postgres?
| samhw wrote:
| Vitess is basically a sharding layer on top of MySQL nodes, so
| yes, it will almost certainly retain whatever MySQL warts you
| dislike. It's clustered MySQL - not simply a different database
| speaking the same dialect, like CockroachDB is to Postgres.
| ComputerGuru wrote:
| Thanks for the explanation! The link for this submission is
| useless and I'm on mobile. I was indeed thinking of
| Cockroach.
| ahachete wrote:
| I'm a bit confused with the "branching" [0] and "non-blocking
| schema changes" [1] features. I'm confused as they sound like
| "the next big thing" and I don't see anything special here. Not
| saying are bad concepts or ideas, the contrary. But not really
| useful either. Surely I'm missing something, so I would love to
| hear from the PlanetScale team here if possible.
|
| I have a strong and long Postgres operational background, so I
| may be also here with assumptions that might be different in
| MySQL/Vitess/PlanetScale. My main concerns/questions are:
|
| * I can't imagine testing DDL changes _without data_. Having data
| there is so important to understand the change and its impact,
| that I won 't do them without data. And unless I'm mistaken,
| these branches only contain DDL, no data at all ("data from the
| main database is not copied to development branches").
|
| * While it sounds neat, has a web UI and a CLI, managing branches
| of a schema and using CI and approval lifecycle... is something
| that sounds like I could do, and possibly better (as it is more
| integrated with tooling and workflows) from Git platforms
| themselves, isn't it? I could do branches, merges, CI, comments
| on MRs, approval... I could even easily build a deploy queue
| ("promote") with a CI. Doesn't sound like too hard.
|
| * I don't understand how the "safeness" and the non-blocking
| nature of changes are ensured. Many DDL changes will take
| different amount of locks on rows or tables, which may cause some
| queuing and even lock storms in the presence of incoming traffic.
| Without incoming traffic, they may run fine. In other words: the
| impact of a migration can only be determined in combination with
| the traffic hitting production. How does PlanetScale do this? How
| for example is handled the case where a DDL changes the type of a
| column to another type which causes a table rewrite, which
| essentially locks the table and prevents concurrent writes?
|
| Again, not saying both concepts are bad. Terminology and
| methodology may be already an innovation. And surely I'm missing
| a lot. But other than this, I don't see myself using this
| (testing migrations without data is a showstopper, and not the
| only one) and I don't see much of an innovation from a safeness
| perspective here.
|
| Why this system isn't one where thin clones of the database are
| created as the branches (e.g. like in Database Lab Engine [2]),
| where you can play with data too, and then some data
| synchronization is performed to switch over to the branch once
| done (is not easy at all, but doable with many precautions)? That
| would be a significant improvement in the process, IMHO.
|
| [0]: https://docs.planetscale.com/concepts/branching
|
| [1]: https://docs.planetscale.com/concepts/nonblocking-schema-
| cha...
|
| [2]: https://postgres.ai/products/realistic-test-environments
| herodotus wrote:
| I am having trouble understanding what a "Serveless Database" is.
| When I do a search of the term, I get hype, not a definition. For
| example:
|
| "What is Serverless Database?
|
| Serverless Database is a prerequisite for Serverless Computing.
| These are specially designed for the workloads which are
| unpredictable and can change rapidly. What's more? This allows
| you to pay only for the database resources you use, on a second-
| by-second basis. ..."
|
| I would really appreciate it if someone would give us a clear
| definition of this term.
| leros wrote:
| As I understand it, they've extracted the storage and compute
| parts of the database and are running them in a scalable way
| such they can automatically add more compute or storage as
| needed.
| mupuff1234 wrote:
| "Serverless" just seems to be the trendy rebranding of "as a
| service".
| moralestapia wrote:
| You don't provision and manage the infrastructure associated
| with your DB. Someone else does it for you and you just focus
| on querying/storing the data and whatever else your business
| requires you to do.
| zild3d wrote:
| Not provisioning & managing is part of it, but the pricing
| model is important too. e.g. RDS and Mongo Atlas are managed
| database services so you're not provisioning and managing the
| infra, but you are paying for dedicated machines and their
| sizes, etc.
| redwood wrote:
| Atlas offers serverless today
| jedberg wrote:
| RDS still requires you to provision servers even if you
| aren't running them.
| brazzledazzle wrote:
| I think Aurora Serverless may be the one exception to
| that.
| moralestapia wrote:
| >but you are paying for dedicated machines and their sizes
|
| I see this as provision & managing as well, but yes I see
| your point. Ultimately it's about the user only caring
| about its data and nothing else (to the extent possible).
| soamv wrote:
| There is no clear definition with universal agreement. It's a
| hype-y term applied with... varying levels of rigor.
|
| However, roughly speaking, "serverless" rolls together 3
| features:
|
| 1. Fine grained pay-per-use (e.g. pay for a query by the number
| of rows scanned)
|
| 2. The pricing dial goes down to zero when usage is small
| enough.
|
| 3. You generally don't control VM/instance-level scaling but
| something closer to the abstraction level of the product being
| claimed as "serverless". For example in planetscale you get no
| control over how many mysql instances actually run your
| queries. This is great for reducing operational complexity but
| not so great for controlling performance. Performance tends to
| be quite opaque -- for example there's nothing I can find in
| Planetscale's docs about latency and throughput. The
| operational benefits are real, though. It's a tradeoff.
| herodotus wrote:
| Thanks
| DeathArrow wrote:
| Probably it can be defined to be "database as a service".
| unfunco wrote:
| Nothing to manage, and it should scale to zero cost when not in
| use.
| 256DEV wrote:
| Basically I understand the idea of serverless in this context
| as abstracting away everything related to managing &
| maintaining the database - i.e. anything like VMs, containers,
| OS, DB processes etc.
|
| So you can set up a PlanetScale MySQL DB and use it just like a
| normal instance of MySQL, but also keep adding data from one
| small set of records all the way up until you have gigantic
| petabyte volumes of data without having to do anything beyond
| sending the data through your MySQL connector.
|
| In theory it should just keep working in a performant way from
| 100 user records for your new startup to the scale of running
| parts of Slack. No choosing bigger and bigger AWS RDS instances
| as you scale, no need for autoscale strategies in case of
| traffic surges or worrying about replicas for perf etc. etc.
|
| As someone who is honestly quite resistant to parts of the
| serverless paradigm this offering actually appeals to me. I
| prefer running my own fleet of VMs and traditional PHP/Nginx
| type stack but have already moved to AWS RDS to abstract away
| some of the replication complexity required to achieve high
| availability DB with minimum hassle. This seems like the
| logical next step and despite being allergic to kind of hype
| you mention finding this is something I'd definitely try out
| before moving other parts of my infrastructure onto anything
| like Lambda.
| zild3d wrote:
| With a dedicated Mongo DB cluster (server-full), you are paying
| for a certain cluster size, per hour. It doesn't matter if you
| read or write any data to it. You're paying for a machine with
| a specific amount of storage capacity and cpu. Use it or lose
| it.
|
| DynamoDB (considered serverless), you're charged based on the
| read and write throughput, and how much you have stored (GB-
| month). If you don't store any data, you're not being charged
| for unused storage capacity, like a dedicated cluster. You
| don't think about the instance size, amount of memory, etc.
|
| - https://www.mongodb.com/pricing
|
| - https://aws.amazon.com/dynamodb/pricing/on-demand/
| pm90 wrote:
| "Pay for only what you use" database.
| freen wrote:
| Scales up and down automatically.
| mbesto wrote:
| I'm curious about what kind of workloads people would be
| using a serverless DB for?
|
| If I understand it correctly, let's say my DB is only being
| used to process 1M transactions from 9am-1pm, I'm basically
| only paying for the load during that time, versus a managed
| DB where it's being paid to be on 24/7. With most serverless
| there is a penalty though - cold startup.
|
| So is it purely an economic play for esoteric DB workloads?
| If my DB is realistically going to be churning for 24/7
| anyway, why would I ever use serverless DBs?
| spelunker wrote:
| Lower operational burden - if you're using a serverless db
| maintaining servers is one less thing to think about. Same
| with any infrastructure - you don't have to worry about
| orchestration, load balancing, authentication, etc.
| mbesto wrote:
| I mean, I can essentially do this with any PaaS DB, no?
| RDS, Azure SQL, Cosmos, etc.
| topspin wrote:
| "Serverless Database"
|
| It's just a database service. You don't run the servers. You
| pay someone to do that and you just connect to it and use it
| and someone else maintains the servers, storage, scaling,
| backups, patches, etc., according to some SLA and other terms.
|
| There are indeed servers somewhere. "Serverless" is misleading
| cloud speak for the otherwise easily understood concept of a
| service.
| alvarlagerlof wrote:
| They shut down when not in use.
| romero-jk wrote:
| DyanmoDB is truly serverless, they give you a http endpoint and
| that's it, you don't have to deal with database connections.
| Xorlev wrote:
| "built on top of the only open source database proven at hyper
| scale, Vitess."
|
| Vitess is a sharding layer. Furthermore, I'm sure Cassandra (for
| all it's flaws) would beg to differ, Netflix had a huge C*
| deployment.
|
| Is hyperscale larger than webscale? Hmm...
| ksec wrote:
| >Is hyperscale larger than webscale?
|
| I am sure Youtube is larger than Netflix, by many order of
| magnitude.
| atombender wrote:
| Note that YouTube no longer uses Vitess. It was migrated over
| to Spanner 2 years ago (maybe earlier).
| sergiotapia wrote:
| Warning: One misconfiguration of your database and you will get
| hammered by billing.
|
| One guy forgot to put foreign keys and got a $20,000 bill.
| sudhirj wrote:
| Indexes, rather. They charge by rows read, so if you don't add
| indexes you will (very quickly) read all the rows in your table
| on every query. That racks your bill really fast.
|
| The foreign key thing is because the author thought foreign
| keys add indexes, but here foreign keys are no-ops and don't do
| anything. Because it's so distributed there's no support for
| foreign keys.
| samlambert wrote:
| Nonsense. It wasn't 20k and it was corrected.
| hyuuu wrote:
| link?
| ralusek wrote:
| Priced at more than a dollar per GB per month? Seems kind of
| nuts.
| cyounkins wrote:
| I'm skeptical that YouTube would be using this for anything
| remotely significant given Google's strong propensity to only use
| internal technologies that they fully control.
| qaq wrote:
| This was opensourced by google founders were @ google and wrote
| the thing
| danielrhodes wrote:
| I doubt YouTube is using the cloud service. If they're not 100%
| on Google's services by now, they still use Vitess. Vitess is
| the open source project PlanetScale manages which lets you
| horizontally scale MySQL databases.
| romero-jk wrote:
| They swithed to spanner.
| rubyn00bie wrote:
| PlanetScale is probably the only database vendor I like, and I
| don't just like 'em. I love 'em. It works extremely well (Vitess)
| and scales easily using PlanetScales UI. The the thing that I
| like about 'em so much is their great customer service... there
| have been a few hiccups but they've always been quick to remedy
| and above all honest about it.
|
| Vitess, for what it's worth, is probably still under the radar
| for most people but is a really nice piece of technology
| (distributed database built on MySQL).
| mooreds wrote:
| If you don't know what PlanetScale is, it is a serverless
| database offering. It is compatible with MySQL and offers lots of
| tooling on top of it. More here: https://docs.planetscale.com/
|
| When I looked at it a few months ago, it had some limitations
| (lack of referential integrity, MySQL version compatibility) but
| I'm not sure if they've been addressed (or, in some cases, if
| they are possible to address).
|
| Anyway, always nice to see more competition in such a
| fundamentally important space, so congrats!
| the__alchemist wrote:
| Where is the data hosted? Is it software to run it locally?
| jjtheblunt wrote:
| the page planetscale.com says their cloud or yours.
| mercwear wrote:
| Thanks for this. The site does a poor job of explaining what
| they actually do
| epberry wrote:
| The referential integrity piece (otherwise known as Foreign
| Keys) will not be addressed and is a limitation - some might
| say feature - of Vitess. They wrote a pretty good piece about
| it here, https://docs.planetscale.com/tutorials/operating-
| without-for...
| mooreds wrote:
| Thanks, that looks pretty useful.
|
| To me, this means that planetscale (and vitess) immediately
| become less useful for a large class of applications:
| existing apps that use an RDBMS that you want to scale.
| Because most existing apps will expect FKs. I haven't tested
| my employer's application (which has FKs) to see what will
| happen, but I doubt it will be smooth sailing.
|
| But there's probably plenty of greenfield development where
| teams would prefer to stick with standards like SQL rather
| than a proprietary noSQL language, and that seems like a
| great fit for this solution.
| mcronce wrote:
| It's not entirely accurate to say that Vitess "doesn't
| have" foreign keys. It doesn't enforce FK constraints
| cross-shard, but it can use FKs to group referential data
| together within a shard, and the MySQL backing that shard
| can enforce FK constraints.
| jeffbee wrote:
| There is not and will never be a solution to the problem of
| horizontally scaling an application and database designed
| jointly for a single-box ACID RDBMS. You have to compromise
| _something_ to get the scaling.
| pier25 wrote:
| What is the compromise of Cockroach DB?
| AYBABTME wrote:
| I believe they use some statistical analysis and declared
| locality tags (from config) to group related cells of
| data on the same nodes and regions, but I don't know how
| effective that proves. Pure performance isn't a strong
| suit of CRDB as far as I can tell, but sometimes that
| works fine, and it does scale well horizontally.
| romero-jk wrote:
| Heavy toll on performance and is not really postgres. For
| the same workload pg can be up to 30x faster or more than
| cockroach. At some point of course, pg won't be able to
| keep up and the horizontal scaling of crdb will beat it.
| jpgvm wrote:
| Depends on your definition of horizontally scaling and
| the inherent size limitations you would be willing to
| accept.
|
| CitusDB for example handles this by allowing FKeys
| between tables that share the same partition key. However
| it's also not -as- scalable as Vitess or as
| georeplication friendly but for many uses that need "more
| than a single box" is probably enough scalability.
| DeathArrow wrote:
| What about CockroachDB and YugabyteDB?
| spookthesunset wrote:
| > You have to compromise something to get the scaling.
|
| And the reason why is basically "physics". The speed of
| light is pretty damn slow all things considered.
|
| Until we find ways to make all the servers in the cluster
| have latency measured in something like picoseconds,
| something is gonna have to give.
| [deleted]
| [deleted]
| danielvaughn wrote:
| I wonder how this stacks up against CockroachDB.
| tonyhb wrote:
| To the best of my knowledge the scaling behind distributed
| ranged KV stores (which cockroachDB uses as its storage
| engine) is easier to manage overall. Whether that's important
| for "serverless" is another question.
|
| I'm sure PlanetScale will help with rebalancing and re-
| sharding but, just like FoundationDB, CockroachDB "promises"
| to automatically re-shard hot keys and ranges for you under
| the hood, and so _in theory_ scales well for a distributed
| read/write heavy system over many nodes with potential
| hotspots. My CockroachDB knowledge is based off of research
| into FoundationDB, but I think it shares many of the same
| underpinnings (even the record layer concept). And
| FoundationDB scales very well.
|
| Vitess, on the other hand, manages the cluster for you but I
| don't think that the rebalancing and sharding is as easy to
| manage. Would love to learn more as I don't know much about
| Vitess.
| harshit164 wrote:
| Vitess using range based sharding and with a single command
| of `Reshard` you can split a range to n sub-ranges.
|
| Ref for how command looks like https://vitess.io/docs/user-
| guides/configuration-advanced/re...
| williamtwild wrote:
| I would love to try this out but having only worked with MSSQL
| and Postgres for any large datasets I wanted to check size
| limitations of MySql and found this from 2020:
|
| "In addition, a practical size limit on MySQL databases with
| shared hosting is: A database should not contain more than 1,000
| tables; Each individual table should not exceed 1 GB in size or
| 20 million rows; The total size of all the tables in a database
| should not exceed 2 GB."
|
| This seems to be parroted elsewhere.
|
| 2GB is nothing. Is this accurate?
___________________________________________________________________
(page generated 2021-11-16 23:01 UTC)