[HN Gopher] The database servers powering Let's Encrypt (2021)
___________________________________________________________________
The database servers powering Let's Encrypt (2021)
Author : alexzeitler
Score : 129 points
Date : 2023-09-16 16:16 UTC (6 hours ago)
(HTM) web link (letsencrypt.org)
(TXT) w3m dump (letsencrypt.org)
| KronisLV wrote:
| > We currently use MariaDB, with the InnoDB database engine.
|
| Oh hey, it's not often that you hear about MySQL/MariaDB on HN,
| so this is a nice change. For what it's worth, it's a pretty
| decent database for getting things done, even if not as advanced
| as PostgreSQL in some respects.
| hyc_symas wrote:
| Still an order of magnitude slower / less efficient than
| OpenLDAP / LMDB.
| berkle4455 wrote:
| Would their needs be met by a simple key-value store?
| WJW wrote:
| Still completely sufficient for the workload, apparently.
| lelouch11 wrote:
| The same argument came up in a big company, incidentally for
| the same use case, certificate store in a CA. This led to
| benchmarks, where OpenLDAP was significantly slower, like
| three orders of magnitudes. Databases have gotten really
| faster in the last couple of years while OpenLDAP has
| stagnated.
| turtles3 wrote:
| I think it is also important to acknowledge that there are
| things innodb does better than Postgres, eg. For most workloads
| an undo log is a far better data structure than implementing
| MVCC by duplicating rows. Autovacuum and vacuum can be an
| absolute nightmare, plus the extra disk traffic the duplication
| generates. Maybe one day OrioleDb will bring this to postgres
| too.
| PaywallBuster wrote:
| Is there an update to this article?
|
| Since the article was written: - AMD has gen4 available - Genoa -
| there's NVMe drives 15tb capacity each
| mcpherrinm wrote:
| The same hardware is still in use today, and I expect will
| continue to be used for some time.
|
| The biggest thing that's happened in the meantime is reducing
| load on the databases by pushing some of the OCSP load out to
| redis caches:
| https://letsencrypt.org/2022/12/15/ocspcaching.html By not
| storing OCSP in the MariaDBs, we reduced write volume and
| storage size significantly.
|
| The next big thing is going to be sharding the database so
| multiple servers can share the write loads, and some queries
| moved out to redis caches.
|
| (I work for Let's Encrypt)
| rz2k wrote:
| And, it looks like with PCIe 5.0 x4 performance exceeds
| 2,500,000 IOPS reads, and 14 GBps sequential read rate.
| tyingq wrote:
| I would guess they try to get 3-5 years out of a new server,
| and this was done in 2021...so still a bit early for a refresh.
| The servers prior to this one had E5-2650 processors, so they
| would have been bought sometime between 2013-2016. Meaning
| their last refresh cycle was at least 5 years.
| wmf wrote:
| Those 2021 servers will probably last them 4-5 years so they
| haven't upgraded yet.
| sgarland wrote:
| > We increase target & max IOPS above the defaults. We still use
| conservative values to avoid excessive SSD wear, but the defaults
| were tuned for spinning disks: innodb_io_capacity=1000,
| innodb_io_capacity_max=2500.
|
| I'd be interested to see if they actually needed this. Those
| parameters affect the baseline and burst IOPS InnoDB is allowed
| to use for background tasks like flushing logs, respectively, and
| generally you don't need to raise them. They default to 200 and
| 2000; those defaults been perfectly adequate for me on a MySQL
| 8.x instance serving 120K+ QPS.
| nologic01 wrote:
| I bet if one could develop an index of useful computation per kwh
| (or $) this project would be near the top.
|
| Imagine if that kind of hyperleveraged impact was the norm rather
| than the exception.
| dmurray wrote:
| It's kind of hard for it to be. If the only software projects
| that existed were the elite hyperefficient ones, the ten guys
| who ran them could just meet up every year for a key signing
| party - no need for a scalable service to automatically issue
| them security certificates.
|
| Computers being commoditized means the median project is very
| low-value, but it also means there are millions of cool
| projects out there.
| jimaek wrote:
| Refreshing to see big projects running bare metal rather than
| overpaying for AWS. Especially one relying on donations and
| sponsors.
| distract8901 wrote:
| Interesting that their previous server is just one or two models
| up from the server I recently grabbed foe $150 on eBay.
|
| I'm just now getting serious about a homelab and it's _shocking_
| how much compute you can get for peanuts from enterprise surplus.
| I 've had this machine (1U poweredge r620) for about a year and
| I'm already itching for an upgrade. Hopefully something with a
| lot of 3.5" drive bays for long-term data hoarding.
| dorfsmay wrote:
| It's because if you don't use that compute power, it's just an
| expensive loud heating system that's going to run up your
| electricity bill!
| sgarland wrote:
| I have three R620s. They are shockingly quiet after being
| booted, even under pretty decent load. If you tune C-states
| for energy conservation in the BIOS, they idle at just under
| 100W each. With moderate load (like you'd have in a homelab),
| around 125W. According to iDRAC, the peak they've seen is
| about 350W.
|
| For better apples-to-apples, each of them has 2x E5-2650v2,
| 3x SATA SSDs, 8x 16 GiB PC3-12800R sticks, and 1x NVMe SSD
| (mounted in a PCIe adapter).
|
| As a comparison, I also have a Supermicro X11SSH-F with a
| bunch (9) of spinning disks. It has a single E3-1220 v5, 1x
| NVMe SSD, 2x 32 GiB PC4-17000U, and averages 125W. Given the
| relative power consumption of HDDs, that's a pretty
| impressive difference IMO.
|
| My entire rack (the aforementioned servers, and some Unifi
| networking gear) pulls around 500-600W in total. When the
| backup server (Supermicro X9DRi-LN4F+; dual-socket something-
| or-other Xeons, some DDR3 ram, and 8x spinning drives) kicks
| on daily to ingest ZFS snapshots, the total consumption is
| around 700W, I think. Maybe 750W.
| d_silin wrote:
| Shows how much bang for buck you can get when hosting your own
| hardware.
| sho wrote:
| I bet that compared to an equivalent load hosted on AWS, that
| lovely box pays for itself in full every month, if not every
| week...
| d_silin wrote:
| Absolutely.
|
| Never understood why people are so infatuated with "cloud"
| options. Yes, it is convinient, but you are absolutely paying
| at least order of magnitude more for the same amount of
| compute/storage.
| dharmab wrote:
| What you pay for is the ability to provision hardware for new
| R&D and projects in minutes rather than days, weeks or
| months. Companies are willing to spend millions a month in
| cloud fees to accelerate hundreds of millions in revenue.
| nunez wrote:
| Because there's no such thing as a free lunch, running your
| own datacenter (or managing datacenter managed services) is
| work, and the clouds are better at providing exactly the
| compute you need when you need it than buying a ton of excess
| hardware you might or might not use (or can't scale quickly
| enough if you _do_ use it all)
|
| A datacenter makes sense if your usage profile is steady
| state and hardly ever changes, or if growth rate is
| predictable and capacity can be procured in advance. Any
| other use case is better suited for the cloud, IMO.
| riku_iki wrote:
| there are several middle points between full cloud and full
| own datacenter: rent dedicated server, rent rack in
| collocation datacenter.
| djoletina wrote:
| You still need to manage the underlying database and
| security which requires a lot of knowledge to do well.
| riku_iki wrote:
| correct, and managed solution will have cost too, quality
| issues: 3p services have bugs and are hacked too and
| that's something you can't control, and added complexity.
| To me it looks like if you have small traffic/storage
| requirements, then cloud may look good. But if you have
| lots of data and need compute then running beefy bare-
| metal server can be very cost beneficial.
| kemotep wrote:
| I still think that you need to manage some security
| things even if you are using AWS.
| civilitty wrote:
| It's gotten absolutely bonkers with all the demand for GPUs
| and the cloud companies are just raking it in hand over fist.
|
| On AWS reserved instances with 8x A100s or H100s (if you can
| even get them) cost more per year than the total upfront
| retail price of the equivalent pods from Lambda Labs. The on-
| demand price is even more absurd.
| qwertox wrote:
| If you don't need to host something big, where you can get
| away with one good VPS, the cloud has the benefit of offering
| cheap bandwidth.
|
| Once you have the bandwidth at your location and you don't
| need to be present in multiple locations, it's cheaper to
| self-host.
|
| Next step would be colocation, but for a start, using cloud
| offerings is a cheap way to be a part of the internet.
| nacs wrote:
| Cloud bandwidth is definitely not cheap. If anything, it's
| where they rip you off the most.
|
| You can get "baremetal"/dedicated servers from places like
| Hetzner and OVH that give unmetered gigabit connections for
| like $50.
| nmjohn wrote:
| You can not get actual unmetered 1gb/s for anywhere close
| to that. If you start pushing anywhere close to that much
| bandwidth, you will be throttled / have your account
| closed. For example, Hetzner caps your bandwidth at 20TB
| per month.
|
| Additionally, if you are actually pushing close to that
| much traffic, you can negotiate guaranteed commit prices
| w/ AWS that are competitive (especially when you consider
| the quality of bandwidth. I can only get ~100 mb/s to my
| hetzner server because of how bad their peering is. I can
| easily saturate my 1GB connection to anywhere in AWS.)
|
| ---
|
| (Having said that, this does only apply to egress via
| cloudfront. Things like charging for intra A-Z bandwidth
| within the same region is insane, and for many workloads
| may be surprisingly expensive.)
| riku_iki wrote:
| > For example, Hetzner caps your bandwidth at 20TB per
| month.
|
| looks like your information is obsolete by 5 years:
| https://www.hetzner.com/news/traffic-limit
| YetAnotherNick wrote:
| Because cloud infatuation is largely a myth. Every middle or
| large organization does periodic cost analysis and select
| cloud based on that analysis not on convenience. e.g. See
| this: https://aws.amazon.com/solutions/case-studies/emirates-
| case-....
| riku_iki wrote:
| > See this
|
| that looks like some marketing material without any
| specifics..
| landemva wrote:
| In my experience, in-house groups racking and configuring and
| maintaining boxes often become a priesthood of negative value
| bickering over pet machine names.
|
| While cloud overcharges for McD product, I know what I get.
| WJW wrote:
| Not to mention that you get it within minutes of asking for
| it. I've worked at places where the internal bureaucracy
| managed to take double digit weeks to deliver a new
| database instance.
| narag wrote:
| _I 've worked at places where the internal bureaucracy
| managed to take double digit weeks to deliver a new
| database instance._
|
| That wouldn't happen if the internal bureaucracy for
| firing the laggards wasn't so excruciatingly slow.
| JambalayaJim wrote:
| It'a very depressing hearing this. Anything cloud at my
| company takes exponentially longer to setup as we have
| layers and layers of bureaucracy for any data that leaves
| our network.
| WJW wrote:
| The grass is always greener on the other side of the
| fence I guess. :)
| pid-1 wrote:
| I did some napkin math (could be very, very wrong) but this
| server costs ~ 225,000 USD according to Dell's webpage.
|
| AWS does not have a 100% similar VM, but you could have
| something close for ~ 20,000 USD monthly. Not that bad.
|
| However, storage costs alone would be astronomic. Like >
| 100,000 USD / month.
|
| I have no idea how much outbound traffic Let's Encrypt serves,
| but that also could be a quite relevant expense.
|
| OFC I also don't know how much Let's Encrypt pays for energy,
| cooling, operations, real estate, etc... but:
|
| > I bet that compared to an equivalent load hosted on AWS, that
| lovely box pays for itself in full every month
|
| I would not take the other side on that bet
| riku_iki wrote:
| > AWS does not have a 100% similar VM
|
| does it have similar instance in principle: you rent
| dedicated server with lots of ssd attached and with no fear
| that instance will be stopped any moment for whatever
| reason?..
| 10000truths wrote:
| $225K for this sounds bonkers: 24x 6.4TB
| Intel P4610 NVMe SSD = 24 x $310 = $7440 2x AMD EPYC
| 7542 = 2 x $1300 = $2600 2 TB DDR4 ECC RAM ~ $13700
| (estimate from a couple of Google results)
|
| Those add up to something like $25K. Sure, there's also the
| price of the motherboard, chassis, maybe some other
| peripherals like external network cards, assembly + support +
| warranty etc. but that doesn't explain an 800% markup.
| nacs wrote:
| The hardware may have cost more in 2021 when this article
| was written.
| icedchai wrote:
| For a critical system, you should really have two, for HA
| purposes. Do they?
| mcpherrinm wrote:
| Yes. Let's Encrypt has two locations, each of which has
| fully redundant hardware, so that's a minimum of 4. We
| actually have a few more.
|
| (I work at Let's Encrypt)
| sseagull wrote:
| We recently purchased a similar server.
|
| One thing to note is that a VAR (as mentioned elsewhere)
| will knock 75% off the price listed on Dell's website.
|
| Another this is that that is way too cheap for those SSDs.
| Enterprise SAS (not plain SATA) SSDs are a lot more than
| $310. Our 7.68TB drives are about $2k each, but worth it if
| they stay problem free.
|
| Even on Newegg, SAS SSD of that size are $900-2000, so add
| warranty and service on top of that.
| 10000truths wrote:
| > One thing to note is that a VAR (as mentioned
| elsewhere) will knock 75% off the price listed on Dell's
| website.
|
| Makes sense.
|
| > Another this is that that is way too cheap for those
| SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more
| than $310. Our 7.68TB drives are about $2k each, but
| worth it if they stay problem free.
|
| I was able to find these two enterprise-grade NVMe SSDs
| on Newegg:
| https://www.newegg.com/p/2U3-0005-000J8 ($474, 7.68 TB, 1
| DWPD) https://www.newegg.com/p/2U3-000S-000Y8
| ($360, 6.4 TB, 3 DWPD)
|
| Is there some kind of catch I'm missing?
| sseagull wrote:
| I am not too much of an expert on enterprise hardware,
| but those are PCIe interface. I don't know how possible
| it is to rack up 24 of those in a single server (you
| would run out of lanes).
|
| This is something more similar to what is in those Dell
| servers (and there are 24 of them):
|
| https://www.newegg.com/samsung-
| pm1643a-7-68tb/p/2U3-0005-000...
|
| https://www.newegg.com/p/0N7-0133-00003
|
| There is certainly a markup with Dell, but it's sort of
| like a cloud vendor - pay for the warranty and service,
| and be (somewhat) hands off if something breaks.
| riku_iki wrote:
| > I don't know how possible it is to rack up 24 of those
| in a single server
|
| that's the one of points of article: Epic cpus have 128
| lanes for a while, and that's how they upgraded to 24
| NVMEs.
| sseagull wrote:
| Oh yeah, the article. I guess this thread got sidetracked
| on the topic of Dell's pricing :). I wonder how common a
| 24-drive NVMe server is.
|
| I don't know all the ins and outs of SAS vs NVMe. Maybe
| someone else can chime in. I am at the end of my
| knowledge now.
|
| I suppose one benefit is the availability of hardware
| RAID controllers, as hinted in the article. But it does
| seem interesting that NVMe is cheaper than SAS, while
| theoretically having higher bandwidth.
| riku_iki wrote:
| Yeah, I feel SAS is obsolete tech, and will be replaced
| by NVMe everywhere going forward.
| barkingcat wrote:
| Threadripper and Epyc has been smashing the pcie lane
| limit for a while now. That's why Epyc is kicking intel's
| ass in server applications.
|
| My personal workstation at the side of my desk has 6 pcie
| nvme ssd's and I can add 4 more without breaking the lane
| bank.
| CathalMullan wrote:
| > These are expensive servers, crossing into six digits, but
| not $200k.
|
| https://news.ycombinator.com/item?id=25865967
| ttegloma wrote:
| > AWS does not have a 100% similar VM, but you could have
| something close for ~ 20,000 USD monthly. Not that bad.
|
| Is that the on-demand cost, or the reserved cost? For
| comparing to buying a server outright, you should be
| comparing the reserved cost. I'm not sure exactly which
| instances you're looking at to get $20k/mo, but I see some
| instances with 64-128 cores/1-2 TB memory for <10k/month.
|
| For storage, I'm not sure how you're getting >100k... I
| plugged in the highest IOPS I could for io2 volumes for 150
| TB of storage and got 30k/mo. Also worth considering here
| that you don't have to provision all 150 TB up front - you
| could start with 5 TB and increase in size as you grow, for
| example.
|
| Still gonna be hella expensive but all of this changes the
| calculus quite a bit from your estimates.
| Spooky23 wrote:
| Nobody is paying base price for a box like that. I'd probably
| bid them against HPE and pay ~90-100k.
|
| If Amazon looks reasonable for something like this, the math
| is wrong. They're renting boxes at 60-70% margin.
| dharmab wrote:
| I'd also bet money that this has higher disk I/O performance
| than the rough equivalent on Amazon.
| louwrentius wrote:
| I think there are ton of great use cases for the cloud, but
| people should try and think for themselves and decide if their
| circumstances and workload is really a good fit.
|
| A ton of people forget that a bunch of servers across a few
| colocations can pay for itself in months, especially if you go
| for second-hand gear that is dirt cheap.
|
| Again, going (back) to colocating hardware could not be good
| fit. But with modern management tools and datacenter services
| like 'remote hands' I think people should not reject it
| upfront.
| candiddevmike wrote:
| Doesn't mention if they're running Linux/what flavor they're
| running. I personally would be wary of running OpenZFS in
| production on Linux, especially ZFS on root. It has bit me in the
| ass too many times on Debian with an update breaking DKMS and
| rendering my system unbootable.
|
| Also, it's very, very strange/worrying to see no mention of disk
| encryption anywhere in the post or the tuning guide. For a
| company with encrypt in the name, that is responsible for the
| majority of trust on the internet, WTF? That should be
| highlighted in their benchmarking. ZFS supports native
| encryption, MariaDB does encryption, how are they encrypting at
| rest/transit/use?
| DistractionRect wrote:
| Usually you aren't updating production servers unless it's a
| security patch, fixes a problem you have, or adds a feature you
| want/need. Even then, usually you have a test environment to
| verify the upgrade won't bork the system
| Caligatio wrote:
| Given that they're using a HSM (actually several), there's
| really not much that needs protection via FDE. The certs are
| obviously public and the domains are in the transparency log.
|
| On the ZFS note: it's been rock solid for me with Ubuntu but a
| living nightmare with Arch. My Arch update would upgrade the
| kernel but then OpenZFS would semi-routinely incompatible
| resulting in an unbootable system.
| pmarreck wrote:
| I had the same issue. I went from Ubuntu -> Arch -> NixOS
| looking for a distro with well-supported ZFS. Finally found
| one (the last one).
|
| This is the magic line from my declarative configuration that
| ensures I never get a kernel update that is incompatible with
| my ZFS-on-root config:
|
| kernelPackages =
| config.boot.zfs.package.latestCompatibleLinuxPackages;
|
| Been running it for 2 years now; quite happy with it, and
| this is probably my "final distro hop". Once you go
| declarative (and climb the learning curve), you are done with
| everything else I guess.
|
| Put it this way, I can switch to an entirely different window
| manager by changing 1 configuration line. And it works every
| time. I've tried that on other Linux distros, and it always
| borks something or other. I can also change my GRUB theme and
| boot animation via Plymouth, something I would have NEVER
| risked on ANY other Linux distro due to the risk of modifying
| boot configuration... but since it's declarative (and
| validated) on NixOS, I've had no issues (just tweaking). If I
| manage to bork something, which is rare, I just reboot into a
| previous update of my OS and fix it and then try again.
| hamandcheese wrote:
| +1 to this, ZFS with NixOS is quite manageable.
|
| One thing I'm not a fan of though is the guide on the NixOS
| wiki was removed and instead points to this guide in the
| OpenZFS docs: https://openzfs.github.io/openzfs-
| docs/Getting%20Started/Nix...
|
| It does not have much in the way of rationale - I.e. they
| say you need minimum 1GB reserve space but don't say what
| reserve space is and why you need it.
|
| I recall the NixOS wiki used to explain options and trade
| offs in a lot more detail.
|
| Nice tip re: latestCompatibleLinuxPackage! Had not seen
| that anywhere yet (probably because I haven't had any
| issues yet).
| pdntspa wrote:
| Consider giving FreeBSD a shot, ZFS support is excellent
| thaumaturgy wrote:
| Yeah, FWIW my first _production_ ZFS deploy was on
| FreeBSD over 10 years ago. It was rock solid (though low-
| demand).
| curt15 wrote:
| ZFS on Linux is commonly used in HPC these days. E.g.
| https://computing.llnl.gov/projects/openzfs
| shrubble wrote:
| ZFS has been very reliable for me since 2006 or so when I first
| started using it on SPARC hardware with the Solaris 10 beta; I
| assume that since they have a backup server and a primary
| server, they don't update and reboot them both at the same
| time.
| vorpalhex wrote:
| You can always boot off ext4 and then just run data off OpenZFS
| pools. The benefits of booting off ZFS are extremely minimal
| compared to having your working data on ZFS.
| pmarreck wrote:
| > I personally would be wary of running OpenZFS in production
| on Linux
|
| A ton of people in the enterprise have been doing this for
| years without issue; https://openzfs.org/wiki/Companies
|
| > especially ZFS on root
|
| I've been running ZFS on root on NixOS since this excellent
| guide (https://openzfs.github.io/openzfs-
| docs/Getting%20Started/Nix...) for about 2 years. Zero issues.
| (Actually, I see they've updated it, I need to look at that.
| Also, they default to encrypted root. I turn it off, because
| the slight hit to performance and extra risk of recovery
| impossibility is not worth it to me.)
|
| > It has bit me in the ass too many times on Debian with an
| update breaking DKMS and rendering my system unbootable
|
| Well, I think you've found your problem, then. (That also might
| be why FreeNAS, which I also have running as my NAS, switched
| to Linux from Debian when they re-branded as TrueNAS
| Core/Enterprise.) Come over to NixOS, where you can simply
| reboot into any of N previous instances after an update that
| borks something (which almost never happens, anyway, because
| you can actually specify, in your machine configuration, "use
| the latest kernel that is compatible with ZFS"). No USB boot
| key needed. Here's the magic line from my own declarative
| configuration: kernelPackages =
| config.boot.zfs.package.latestCompatibleLinuxPackages;
|
| Aaaaand... DONE. ;)
|
| > Also, it's very, very strange/worrying to see no mention of
| disk encryption anywhere in the post or the tuning guide. For a
| company with encrypt in the name, that is responsible for the
| majority of trust on the internet, WTF?
|
| You're assuming they're not doing it, without evidence. Also,
| if they're already managing the security around their certs and
| cert generation properly, they might not need FDE. FDE is
| overrated IMHO, frankly, and also incurs a performance cost, as
| well as an extra risk cost (try recovering an encrypted drive
| to know what I mean). In short, religions are bad, even in
| technological choices; there is no single technological
| configuration choice that is 100% better than all possible
| alternative configurations.
|
| > That should be highlighted in their benchmarking. ZFS
| supports native encryption, MariaDB does encryption, how are
| they encrypting at rest/transit/use?
|
| Multiple layers of encryption incur an extra performance cost
| with almost no gain in extra security.
| [deleted]
| nunez wrote:
| These performance improvements are absolutely insane, and
| churning out certs at speed is the _perfect_ use case for these
| CPUs.
|
| Kudos the LE team!
| hu3 wrote:
| Indeed. If I had to bet, I would say doubling the memory
| allowed some/all indexes to fit in RAM. RAM is usually a big
| improvement when it comes to Relational Database Management
| Systems (RMDBS).
| tda wrote:
| I recently bought my first server, just about the cheapest Dell
| had available (still EUR2000). I was completely undewhelmed by
| the specs, it even has spinning rust. The upgrade costs to SSD's
| were like 250 per disk. So reading these specs, how much does
| such a server cost? Or do you put the disks in yourself? Can you
| negotiate a 50% discount? So many questions I have as a newbie in
| the entireprise server world.
| barkingcat wrote:
| No one buys dell servers at list price.
| Dachande663 wrote:
| From old knowledge, you're out by about 2 orders of magnitude.
| milesdyson_phd wrote:
| If you are an enterprise (or just big enough) the prices on
| Dell's site are meaningless, they are just conversation
| starters if that.
| dang wrote:
| Discussed at the time:
|
| _The database servers powering Let 's Encrypt_ -
| https://news.ycombinator.com/item?id=25861422 - Jan 2021 (226
| comments)
| gloyoyo wrote:
| Always nice to get an upgrade.
| tyingq wrote:
| They were issuing 2M certs/day when this happened in 2021, now up
| to 3.4M/day.[2]
|
| Looks like it scaled pretty well so far.
|
| [2] https://letsencrypt.org/stats/
| mark242 wrote:
| Someone reading the headline of this might think "oh, they're
| using Cockroach, or Fauna, or Planetscale" -- nope, this is about
| next-gen hardware powering their single-write (with a number of
| read replicas) MariaDB instance.
| riku_iki wrote:
| they probably started building before Cocroach and others
| became solid choice, and now would need some big migration
| project to switch.
| esafak wrote:
| Indeed what I expected. It should have been titled "The Next
| Gen Servers Powering Let's Encrypt's Database"
___________________________________________________________________
(page generated 2023-09-16 23:01 UTC)