[HN Gopher] Migrating Facebook to MySQL 8.0
___________________________________________________________________
Migrating Facebook to MySQL 8.0
Author : moneil971
Score : 281 points
Date : 2021-07-22 17:54 UTC (5 hours ago)
(HTM) web link (engineering.fb.com)
(TXT) w3m dump (engineering.fb.com)
| wewe2fg23 wrote:
| Only oldies uses Facebook in 2021
| andremendes wrote:
| Anyone else puzzled by the shaking migration animation? The
| original file has a 1920x1080 resolution. I wonder if it is
| related to the shaking. Good article nevertheless.
| andrewmcwatters wrote:
| If you look closely, you can see the engineers nervously
| rolling out changes to production, which is where the
| vibrations in the .gif come from.
| Normal_gaussian wrote:
| It has ?w=1024 causing resizing effects:
|
| Shaking: https://engineering.fb.com/wp-
| content/uploads/2021/07/CD21_3...
|
| Not shaking: https://engineering.fb.com/wp-
| content/uploads/2021/07/CD21_3...
| zestyping wrote:
| I see it too. It's weird and distracting -- something must have
| gone wrong in the rendering.
| andrewmcwatters wrote:
| I wonder what order of magnitude of rows they deal with and how
| they optimize indices for querying that data.
| datavirtue wrote:
| Like everyone else?
| andrewmcwatters wrote:
| I didn't downvote you, but you were downvoted for this. So
| maybe I'll explain further, instead, to stir conversation.
|
| As you deal with more and more rows, it becomes imperative
| that all of your where clauses hit indexes/indices, but even
| beyond that, with large enough row sizes, it's not enough to
| provide fast responses.
|
| I suspect they deal with this in the form of some sort of
| caching outside of MySQL, but I haven't read into it.
|
| I'd be curious about how you respond to such challenges at
| the billions or trillions of rows orders of magnitude.
| Millions can be difficult enough, but beyond that you may
| have queries that effectively never return if you do not
| plan.
|
| Related reading:
|
| [1]: https://dba.stackexchange.com/questions/20335/can-mysql-
| reas...
|
| [2]: http://webscalesql.org
| evanelias wrote:
| Facebook is heavily sharded, which keeps the size of each
| physical table at reasonable levels.
|
| Similar story at nearly every large tech company using
| MySQL, aside from some more recent ones that go for the
| painful "shove everything in a huge singular AWS Aurora
| instance" approach :)
| andrewmcwatters wrote:
| Thanks for that insight! Sounds like maybe you've seen an
| interesting thing or two given your background.
| vosper wrote:
| > the painful "shove everything in a huge singular AWS
| Aurora instance" approach :)
|
| I haven't used Aurora - what's painful about this
| approach (apart from potentially being locked-in to AWS /
| Aurora)?
| evanelias wrote:
| Aurora lets you grow your db storage up to 128 TB, but
| things become difficult and slow long before this point.
| This is true operationally (e.g. schema changes,
| restoring from a backup, etc) as well as potentially for
| application / query performance issues.
|
| That said, sharding can be very painful too in other
| ways. But at least it permits infinite horizontal
| scaling, and it makes many operational tasks easier since
| they can be performed more granularly on each smaller
| shard.
|
| Besides, once you get anywhere near Aurora's storage
| limit, you'll need to shard _anyway_. So it really just
| buys you some time before the inevitable.
| ahiknsr wrote:
| > I suspect they deal with this in the form of some sort of
| caching outside of MySQL, but I haven't read into it.
|
| It is handled by TAO,
| https://engineering.fb.com/2013/06/25/core-data/tao-the-
| powe...
| andrewmcwatters wrote:
| Thank you for sharing this!
| fukmbas wrote:
| Lmao FB is on mySQL? Someone fucked up
| markus_zhang wrote:
| >It took a couple of years to complete porting all of these
| features. By the time we got to the end, we had evaluated more
| than 2,300 patches and ported 1,500 of those to 8.0.
|
| This is some tremendous effort as I see. I wonder how many people
| work on these evaluations.
| hitekker wrote:
| Seems like good, old-fashioned engineering work to me. No
| breathless evangelism or careerist claptrap in sight.
| linspace wrote:
| It seems that "move fast and break things" is not for billion
| dollar companies.
|
| My impression too. Very meticulous work, just the facts.
| gigatexal wrote:
| Where can I read more about the custom things FB added to MySQL
| 5.6? Their work on replication and the DDL work seems
| fascinating. Are there scholarly papers, white-papers, blogs?
|
| Are they storing the friend graph in MySQL?
| midom wrote:
| there's also this team page, although it has been somewhat
| lower on volume lately -
| https://www.facebook.com/MySQLatFacebook/
| HarrisonFisk wrote:
| Direct source is available at:
|
| https://github.com/facebook/mysql-5.6
|
| The social graph is indeed stored in MySQL via TAO. A recent
| white paper about MyRocks is:
|
| https://research.fb.com/publications/myrocks-lsm-tree-databa...
| alberth wrote:
| It's interesting to see FB migrating so much over to Oracle.
|
| 2 weeks ago was FB use of GraalVM (from Oracle). And now this.
|
| https://news.ycombinator.com/item?id=27782475
| dcolkitt wrote:
| I guess one could dream of the remote possibility that Facebook
| outright acquires Oracle. It seems like a weird match, but it'd
| give FB a toehold in a lot of verticals that it currently has
| no presence in. Whatever criticisms you have of FB, it's at
| least a pretty open source friendly company.
| amelius wrote:
| What would happen to Oracle's infamous legal department then?
| munk-a wrote:
| Then they'd leverage social media oversharing to find more
| targets to litigate - it'd be a win-win! /s
| aloisdg wrote:
| You mean pretty much the whole company? /s
| rattray wrote:
| Fabulously lavish retirements?
| dylan604 wrote:
| They'll start serving ads and sharing personal data of all
| of the people they sue?
| rrdharan wrote:
| Facebook has no interest in infrastructure / enterprise
| revenue streams (Facebook for Work notwithstanding). See the
| history of Asana, Cloudera and Phabricator (and even Dropbox,
| from an employee/recruiting perspective, kinda).
| hodgesrm wrote:
| The MySQL 8.0 upgrade is not a migration to Oracle. Facebook
| has run MySQL for many years at this point and, as the article
| describes, operates on a private fork. They have employed some
| of the leading engineers in the MySQL community to support
| their version of MySQL.
| maxk42 wrote:
| Oracle has a very complete and much cheaper cloud solution
| relative to AWS. I've begun migrating my own projects over to
| their cloud. The savings on egress costs alone pays dividends.
| mr_overalls wrote:
| I find the comments in this article very accurate:
|
| https://news.ycombinator.com/item?id=15160149
|
| > ORA is the elephant's graveyard of software.
|
| > Once something gets bought by them, you know it is done.
| Slowly, but surely.
|
| > They perform a function akin to the maggots that destroy
| cadavers in nature. Part of the overall ecosystem.
|
| > ORA stopped being a tech co a while ago, now it is a
| finance play. Use cash to buy a business for its locked in
| customers, gut it to squeeze max money out of it until last
| customer is gone. Rinse, repeat.
| adrianmsmith wrote:
| This is not so with Java. They have moved Java along a lot
| since they bought it; they released 1.8 finally, and have
| introduced lots of new features such as type inference
| since then.
|
| MySQL is also going strong after the acquisition. Lots of
| new features.
| BossingAround wrote:
| And yet MySQL is very much alive..? Java has fully
| opensource implementations too, and has had innovations
| like GraalVM and Jakarta EE.
|
| I think I'm too young to hate Oracle with a burning
| passion. I would be very scared of their sales/corporate
| team, but I mean, even having large contracts with Google
| or Amazon is difficult to say the least.
| danpalmer wrote:
| MySQL is alive for companies already using MySQL, but
| would well informed companies be choosing it today? I'm
| not so sure.
|
| I think you either want it to have more functionality and
| you choose Postgres, you want a particular performance
| profile and you pick MariaDB, or you want the enterprise
| support and you pick Oracle's database. Why you would
| pick Oracle-supported MySQL today I'm not sure.
| abledon wrote:
| didn't the new basecamp "Hey" email product choose to use
| MySQL?
| danpalmer wrote:
| I think so, but they have been operating MySQL at scale
| for many products for about 20 years, so I can understand
| why they might not choose MariaDB (or Postgres, Oracle,
| etc).
| ericyan wrote:
| Easy to operate, perhaps?
|
| With MySQL group replication, it is pretty easy to set up
| a HA cluster that allows you to sleep soundly. It can
| also handle thousands of connections without the need of
| a connection pooler.
| danpalmer wrote:
| Sure, but MariaDB is the more obvious choice surely?
| evanelias wrote:
| No, why do you think so?
|
| Among the largest users, MySQL (or Percona Server) is
| chosen far more frequently than MariaDB.
|
| GP mentioned Group Replication, which isn't even
| available in MariaDB.
| threeseed wrote:
| > have more functionality and you choose Postgres
|
| PostgreSQL still after all these years has a terrible
| story when it comes to HA and horizontal scalability.
| Nothing is built in or supported. And one of the only
| companies to provide something half decent i.e. Citus is
| now owned by Microsoft and so that project is now at
| risk.
|
| At least MySQL has something built-in.
| danpalmer wrote:
| So MariaDB then.
| redis_mlc wrote:
| > I think you either want it to have more functionality
| and you choose Postgres
|
| False. MySQL and Postgres are comparable feature-wise.
|
| @danpalmer: you're making an ass of yourself. It's clear
| from your comments that you're around 10 years out of
| date.
| alberth wrote:
| Just curious, so which companies do people consider are
| creating/selling interesting and innovative ENTERPRISE
| software?
| llimos wrote:
| Microsoft. King of the enterprise with no intention of
| losing the title.
|
| PowerApps, Power Automate etc. - while not necessarily
| innovative in a vacuum - are, when you consider they are
| bringing cloud-based automation and RPC to the
| (enterprise) masses.
| voidfunc wrote:
| This is such a hilariously myopic HN take.
| mr_overalls wrote:
| I work for a government agency in a rather poor flyover
| state. We ran Oracle databases and a few other products
| for multiple decades, until Oracle conducted an audit and
| decided to shake us down for everything we were worth. We
| migrated away from their products, but only after being
| forced into expensive multi-year subscriptions for
| products that we didn't need.
|
| I will never forget the arrogance and greed of their
| legal/sales people. They are bloodsucking sociopaths.
| abraae wrote:
| I'm no fan of oracle's audit approach but it sounds like
| you were expecting special leniency to not pay for their
| stuff - for decades - because you are a govt agency/ live
| in a low income place. Not how business works. If you're
| playing by the rules, no one can force you into any long
| term contracts you don't need.
| myohmy wrote:
| I'm also a government Oracle dev. I have no way of
| knowing what I'm licensed for and they don't disable the
| features we're not licensed for. I literally have to talk
| to my director to talk to our account rep to figure out
| that I'm not allowed to use table partitioning.
| abraae wrote:
| Agreed that Oracle licensing is byzantine, probably
| deliberately. But devil's advocate says that any
| organization should be aware of what they are actually
| licensed for, even if that means contracting a third
| party to ensure compliance.
| sly010 wrote:
| Totally off topic, but I found it funny that in the offical
| oracle cloud intro video the robot arm in the background
| misses the box, right as the narrator mentions the advantages
| of automation at 1:10:
| https://www.youtube.com/watch?v=f_BDfMdGn3M&t=61s
| suyash wrote:
| lol good eye! but it shows me how real tech works, nothing
| is 100% guranteed
| wizzwizz4 wrote:
| I think you'll find the box missed the robot arm.
| abledon wrote:
| I hate oracle, but man do i love their art style branding,
| kind of a 'painting' vibe
| topspin wrote:
| > Oracle has a very complete and much cheaper cloud solution
| relative to AWS
|
| Tell us more. Ignore the haters please. I'd love to have your
| insights.
| maxk42 wrote:
| It's all on their site - https://www.oracle.com/cloud/
| Their price list is at https://www.oracle.com/cloud/price-
| list.html and the egress is under the "networking" section.
| Basically they give you 10TB outbound data transfer for
| free and charge as little as $0.0085 per GB after that.
| That's a savings of 90% (or 100% if you use less than 10TB
| transfer, which most apps do)
|
| Additionally their cloud servers (and most other services
| as well) are much cheaper than comparble EC2 instances and
| every one of their services I've used so far also includes
| a free tier so you can try them out without paying
| anything.
|
| All in all pretty pleased so far.
| gunapologist99 wrote:
| Yeah, AWS (and Azure, and GCP) really has notoriously
| expensive bandwidth, and it's the one thing they will
| never budge on in negotiations, since they know it's
| where they have nearly 100% profit margin; you can avoid
| using most services, or use compute etc more efficiently,
| but it's hard to consume less bandwidth!
|
| It sounds like Oracle is actually pricing bandwidth at a
| reasonable price. This is quite a shock, coming from a
| historical perspective!
| luhn wrote:
| That has nothing to do with MySQL. Facebook operates their
| own DCs and hasn't indicated any interest migrating
| elsewhere.
| jmnicolas wrote:
| Tying yourself to Oracle is quite a risky choice in my
| opinion.
|
| After having used their non cloud products 10 years ago, I
| wouldn't do any business with them. At all.
| na85 wrote:
| I'd say the same about Facebook.
| suyash wrote:
| You can see Oracle or you can see Open Source Awesome
| technology they are using, both of these are solid OSS
| projects.
| cranekam wrote:
| It's a stretch to claim this is a "migration". Facebook has
| used MySQL since its very early days -- certainly since before
| Oracle acquired Sun and thus MySQL AB.
| slownews45 wrote:
| Somehow Facebook and Oracle seem like fitting partners. Too
| horrible companies. One fighting the fight with lawyer / sales
| teams, the other with clickbait / outrage
| simonw wrote:
| I'd love to know roughly how many engineers (or engineer hours)
| this took - sounds like a mammoth project!
| schuyler2d wrote:
| I'd be interested more in what features were in their (3):
|
| > Non-MyRocks Server: Features in the mysqld server that were not
| related to our MyRocks storage engine were ported.
|
| i.e. things that they haven't shared with the MySQL community but
| aren't related to the storage layer. Why would/should they keep
| those private? Considering that RocksDB is open-source, clearly
| it's unlikely it's because they're 'trade secrets'.
| HarrisonFisk wrote:
| The code is open source:
|
| https://github.com/facebook/mysql-5.6
| sriram_sun wrote:
| "MySQL, an open source database developed by Oracle" - Gawd! Were
| most devs at FB born after the acquisition?
| evanelias wrote:
| It didn't say _originally_ developed by Oracle. The statement
| is accurate; MySQL is currently developed by Oracle.
|
| At big companies, sometimes the corporate lawyers are overly
| picky about wording in blog posts and public presentations,
| e.g. "make it clearer that MySQL isn't our in-house product"
| type of thing.
|
| fwiw, several of FB's top database folks originally worked for
| MySQL AB, so they're well aware of the development history.
| cmrdporcupine wrote:
| My old-man brain still can't compute this statement: _" MySQL, an
| open source database developed by Oracle"_. I know this is now
| factually true, but late-90s me keeps looking over my shoulder
| and freaking out.
| soperj wrote:
| should read: "an open source database bought by Oracle".
| blantonl wrote:
| The hair on my neck still stands up when thinking about dealing
| with Oracle (and Sun) contracting, purchasing, and sales teams
| in the 90's.
| briffle wrote:
| I dealt with them in 2016. I assure you, not much has
| changed.
| cmrdporcupine wrote:
| Don't forget IBM
| beefield wrote:
| I have managed to find myself accidentally twice at the
| receiving end of IBM "services". Conclusion was pretty much
| same both times:
|
| Never. F###ing. Again.
| silicon2401 wrote:
| What's it like working at IBM? I've seen it get good
| ratings online
| forbiddenlake wrote:
| This question is extremely hard to answer, because IBM
| has ~345k employees in 177 counties. It's anywhere from
| 0/10 to 10/10.
| zerop wrote:
| Agree. I have used IBM web sphere in 2008. IBM, Oracle, Sun
| is a tribe.
| filmgirlcw wrote:
| I had the same reaction! I was like, "technically, yes, but..."
|
| I haven't thought about MySQL proper in years, but it is
| interesting Facebook is moving to MySQL 8 rather than going to
| MariaDB, which as far as I know, had MyRocks built-in.
| dheera wrote:
| I just configured a new router and had to set up my dynamic DNS
| updating service on it.
|
| Went to dyn.com and the goddamn website is gone, redirects to
| some corporate Oracle bullshit and PR, and I can't login to
| that Oracle crap with my dyn.com account and password.
|
| In case any confused people googling for this stumble upon this
| comment you need to go to: account.dyn.com which they don't
| tell you on the Oracle page.
|
| You can log in there with your former dyn.com credentials.
| fredliu wrote:
| I can't read pass "developed by Oracle". It is factually not
| true. Unless there's so much code change after the acquisition
| that there's no single trace of the original open source
| version of MySQL left in there.
| johannes1234321 wrote:
| MySQL always was primarily developed by an commercial entity.
| And always gave an license out under GPL (i believe very
| first used some other license as GPL wasn't known back then
| ...) Oracle took over the development team (lots of pre-
| acquisition staff included) and legal rights to all of that.
|
| (Disclaimer: I started at independent MySQL AB more than ten
| years ago and now wear an Oracle badge, after wearing Sun for
| a while)
| fredliu wrote:
| Maybe it's legal talk that I don't understand, but the fact
| that Oracle owns the majority of the dev team _today_ ,
| doesn't mean Oracle developed MySQL. Because at the time
| MySQL was "developed", Oracle didn't own MySQL, nor did
| Oracle "own" the developers.
| johannes1234321 wrote:
| Oracle still develops and to a notable number the same
| people. Also Oracle steered 8.0 which was a big change to
| internals. Anyways won't argue about single words.
| PeterZaitsev wrote:
| I think MySQL Engineering team does great job with Engineering,
| Do not want to deal with Oracle Sales ? I would invite you to
| talk to Percona for Commercial Support and Services Options
|
| Full disclosure - I'm CEO at Percona :)
| dvtrn wrote:
| _Full disclosure - I 'm CEO at Percona :)_
|
| oh hi. big fan of you all. xtrabackup and PMM are wonderful
| tools
| PeterZaitsev wrote:
| Thank you!
| zepearl wrote:
| I got sad when I read that TokuDB in Percona & MariaDB was
| being dropped :(
|
| Replaced TokuDB with MyRocks but it's totally different =>
| now finally managed to have good ~constant performance
| insert&deletes (had to test & tune a lot), but "updates"
| remain a problem (they write a lot as I understand that any
| update always has to rewrite the entire row).
|
| Using InnoDb instead of MyRocks is NOK for me (problems with
| deletes and related hole punching + bad that tables can never
| shrink when stuff gets deleted), I don't see other
| alternatives... .
|
| I'm now experimenting with other DBs (right now with
| CockroachDB but it doesn't seem to be very stable, might then
| try as well Postgres but I'm scared by the lack of SQL-hints
| to cover worst case scenarios).
| jtbayly wrote:
| Agreed. And in one sense (English is complicated) it's not true
| in its most obvious interpretation--that Oracle is the reason
| it exists. It was developed by others. Changing it to
| "currently being developed by Oracle." It's not that "developed
| by" can't mean "currently being developed by." It just isn't
| the most obvious meaning.
| pmarreck wrote:
| can it do DDL commands in a transaction yet?
| jjeaff wrote:
| I always see people asking about this. But why? Are you making
| schema changes that frequently?
| handrous wrote:
| Schema changes are pretty much the riskiest damn thing you
| can do with a database, so it's natural people would be
| _extremely_ interested in being able to confine as much of
| that as possible to transactions, even if changes are rare
| (which, arguably, they 're rare in no small part _because_
| they 're so risky).
| simonw wrote:
| It's my opinion that schema changes should be cheap, fast and
| unexciting.
|
| Unfortunately, in my experience once you are operating at
| scale they are expensive, slow and fraught with peril.
|
| The result is that engineering teams often chose to build
| poorly designed schemas and take on technical debt rather
| than make a schema change to a core table. This is a terrible
| anti-pattern!
|
| I'm massively in favour of anything that makes schema changes
| cheaper and more boring. I really enjoy working with the
| Django schema migration system for this reason - though it
| won't help you as much if you need to make schema changes to
| older versions of MySQL (I've not tried schema changes
| against 8.0 yet) since you'll likely still need to use GitHub
| ghost or Percona pt-online-schema-change.
| gfody wrote:
| yeah treating every schema change as a db migration might
| be our industry's worst anti-pattern. transactional ddl
| allows you to do stuff like: begin tran
| ..refactor data, drop/create tables.. ..create
| backwards-compatible views.. commit
|
| ..against a live database without risking consistency
| spullara wrote:
| Before Twitter migrated off their MySQL user database there
| were many columns that had been repurposed for exactly this
| reason.
| simonw wrote:
| I've been contemplating adding a JSON column to all of my
| tables just to provide somewhere that covers those cases
| - still controlled with source-code level validation and
| with the expectation that any column which needs to be
| used for filtering rows gets promoted to "real column"
| status.
|
| I'd much rather use a robust, performant real schema
| migrations tool though.
| yxhuvud wrote:
| Yes? Also, I want them to not be able to break halfway.
| pauldino wrote:
| No, but MySQL 8 does at least have atomic DDL now, as well as
| instant DDL for some cases (like adding a column).
| adrianmsmith wrote:
| In terms of more fundamental features: can it do CHECK
| constraints yet?
| mrcarruthers wrote:
| If I remember correctly, 8.0 has them finally. 8.0 seems to
| have fixed a lot of the weird legacy eccentricities of MySQL
| mrcarruthers wrote:
| Reading the article, seems like facebook has added support for
| it in one of their internal patches.
| spullara wrote:
| Nope. In terms of broad usage I think only Postgres supports
| it.
| gfody wrote:
| sql server and db2 also support ddl in transactions, oracle
| still does not
| mackman wrote:
| Wow. I worked on the upgrade from 5.0 to 5.1 (I think). We had
| fewer than 100 patches at the time. 1700 sounds like a nightmare.
| I apologize for any and all of the patches I wrote.
| FpUser wrote:
| >"MySQL, an open source database developed by Oracle, "
|
| Sorry but there is a difference between "developed" as in
| originally created and "being developed" which is the case here
| in relation to Oracle.
| bratao wrote:
| +1 for MyRocks. If you can, try it. We store some text data and
| the compression is over 10 times better than InnoDB with a
| significative speedup.
| ksec wrote:
| MySQL 5.6 was released on 5 February _2013_.
|
| MySQL 8.0 was released on 19 April _2018_.
|
| >The 8.0 migration has taken a _few years_ so far. We have
| converted many of our InnoDB replica sets to running entirely on
| 8.0.
|
| At the scale of Facebook I wonder if they are the largest MySQL
| user on the planet.
|
| And I take this opportunity to ask, does anyone know how does the
| MySQL roadmap works? What sort of features are coming or when is
| 9.0 expected to arrive.
| Thaxll wrote:
| Youtube also use MySQL.
| dheera wrote:
| Wow. Why don't they use in-house Google databases, which are
| supposedly vastly more scalable than MySQL?
| SQueeeeeL wrote:
| Probably because a lot of those benefits don't actually
| translate that well towards their problem space. These
| aren't a bunch of 20 year olds throwing together a MongoDB,
| they're professionals using tools to actually make a
| product
| williamdclt wrote:
| I don't think the people behind Spanner are a bunch of
| 20yo throwing together a MongoDB
| jacquesm wrote:
| Because there is a substantial cost to such migrations. If
| it works don't fix it.
| mastersummoner wrote:
| While "if it ain't broke, don't fix it" is often a
| perfectly valid approach, there are going to be times
| when an upgrade can result in significant performance
| improvements or fewer admin headaches.
| linspace wrote:
| Does anything "just work"? Everything is constantly
| changing, new features always in demand, more users, more
| data.
| lazide wrote:
| And you need to support that for anything you have ON TOP
| of any migration work - which is huge, and also a
| similarly moving target
| myohmy wrote:
| If IT is on the value chain this seems insane. If IT is
| ancillary to the value chain then this is standard.
| vp8989 wrote:
| YouTube built Vitess, which lets you scale out MySQL
| writers. If I can scale out both my writers and my readers
| then what could be "vastly more scalable" than that?
| WJW wrote:
| As far as I can tell, Vitess still has trouble with
| cross-server transactions and distributed locking.
| Presumably something "vastly more scalable" could work
| around this.
|
| The Spanner team noticeably put out
| https://cloud.google.com/blog/products/databases/inside-
| clou... a few years back, which tl;drs that they managed
| to get a reliable enough network that partitions don't
| really happen anymore and so you can get both C _and_ A
| from the CAP theorem. I don't think that Vitess is quite
| at that level, especially since you can self-host it and
| thus the devs don't have such network-level uptime
| guarantees.
| vp8989 wrote:
| Are cross-server transactions scalable? Would anyone
| architect a system that did them en masse unless they
| absolutely 100% had to?
|
| I am just speculating here but I doubt that YouTube
| migrated to Spanner so that they could do cross-server
| transactions. I would think consolidating the Ops burden
| into the GCP org and also serving as a trophy "customer"
| were higher on the list of reasons. But again, just
| speculating...
| ksec wrote:
| I guess people from PlanetScale will know. Not sure if
| they are reading the thread.
| ansible wrote:
| Spanner paper:
|
| https://static.googleusercontent.com/media/research.googl
| e.c...
|
| It is designed from the ground-up to be globally
| distributed. Some other databases like FoundationDB (now
| owned and killed? by Apple) and CockroachDB were designed
| to scale across multiple datacenters.
|
| _Correction:_ FoundationDB is alive as open-source
| again, yay!
| svieira wrote:
| FoundationDB was re-released as open source a couple of
| years ago.
|
| https://www.foundationdb.org/
| imoverclocked wrote:
| AFAIK, FoundationDB is alive and well. What makes you
| think it's killed?
| capableweb wrote:
| It was acquired by a huge company that sometimes kills
| products. Shortly after the acquisition, the downloads
| were removed from the website. It did go open source
| "recently" though which made it longer lasting, but I'd
| understand if someone would think it was dead by now.
|
| - https://news.ycombinator.com/item?id=9259986 - Apple
| Acquires FoundationDB - March 24, 2015
|
| - Comparison between March 25 and March 14, 2015 - https:
| //web.archive.org/web/20150314231702/https://foundatio...
| (14th March) > https://web.archive.org/web/20150325003252
| /https://foundatio... (25th March)
| gresrun wrote:
| YouTube _used_ to use a MySQL-based system called Vitess[0].
| It no longer uses it and has migrated to Spanner.
|
| [0]: https://vitess.io/
| annexrichmond wrote:
| was that migration a technical decision or a political one?
| Thaxll wrote:
| Thanks for the update! I found more details there:
| https://opensource.google/projects/vitess
|
| > How Google uses Vitess
|
| > Vitess was serving all YouTube database traffic from 2011
| to 2019.
| wikibob wrote:
| Ah this is interesting. Do you have any references on their
| migration to Spanner?
| buro9 wrote:
| gresrun works at YouTube, if by reference you mean
| citation then his comment is it.
| jjnoakes wrote:
| wikibob was probably looking more for public
| documentation about the why and how rather than
| confirmation that the statement is true.
| owaislone wrote:
| Probably because Youtube was using MySQL before being
| acquired and switched to a better, probably fully managed
| DB that was much easier (free?) to maintain and scale.
| [deleted]
| dekhn wrote:
| google internally has always had a split mind about mysql. It
| never really jibed with the underlying google architecture
| but played an important role in multiple products (at least
| partly due to its replication features). Replacing it
| completely has turned out to be a huge challenge. I gained a
| fair amount of respect for mysql after seeing it applied at
| scale, but it was also a bit scary.
| MuffinFlavored wrote:
| > At the scale of Facebook I wonder if they are the largest
| MySQL user on the planet
|
| I have to imagine they use other databases as well, right?
| We're much smaller than Facebook and we have some Mongo and
| some Postgres and a mix of Go/Rust/Scala/node.js.
|
| I wonder how many boxes they have dedicated to be being MySQL
| machines. I wonder how big their tables are and how many writes
| per second they see during the day.
| fowl2 wrote:
| GitHub also seems to use MySql, as recently as 2020[1]
|
| [1] https://github.blog/2020-05-20-three-bugs-in-the-go-mysql-
| dr...
| tpetry wrote:
| So there has not been any major improvement in MySQL for 3
| years? There hasn't been released any minor version? Can
| someone explain the current state more?
| PeterZaitsev wrote:
| No. MySQL 8 changed its engineering approach with features
| released in minor releases.
|
| In this regard it is like MacOS X or Windows 10, whatever you
| prefer where features are shipped without major version
| change
| tpetry wrote:
| Sure, and this is great. But The newest release is 8.0.26,
| so there hasn't been any minor release yet - in 3 years.
| And i can't find anything about new MySQL features or
| planned features except the big 8.0 release.
| PeterZaitsev wrote:
| There are new features introduced in 8.0.26 as in .25 .24
| etc before that check out release notes.
| fipar wrote:
| There have been several minor releases in the past 3
| years: https://dev.mysql.com/doc/relnotes/mysql/8.0/en/
| taywrobel wrote:
| In semantic versioning parlance those would all be
| considered "patch" releases, not a "minor" release. The
| minor version segment in a semantic version is the 2nd
| number, so most people would consider the next "minor
| release" for MySQL to be 8.1.0.
|
| If following the guidance set out at https://semver.org,
| a patch release wouldn't add any new functionality, it
| would just address bugs in a backwards compatible way. A
| minor version would introduce new functionality in a way
| that doesn't violate backwards compatibility.
|
| Quickly scanning the release notes for some of the MySQL
| 8 versions, they seem to be generally sticking to this;
| most of the versions fixed some bugs or unintuitive
| behavior, but it doesn't look like much new functionality
| has been introduced since 8.0.4, the last release
| candidate for the major version bump.
| PeterZaitsev wrote:
| MySQL does not follow semantic version and there have
| been features introduced in minor released including some
| incompatible changes as you can see for example https://d
| ev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-24....
| Tostino wrote:
| Yeah, no thank you. I'll stick with Postgres.
| evanelias wrote:
| Many major new features have been introduced in 8.0 point
| releases. Some examples:
|
| * 8.0.12 added INSTANT algorithm for adding new columns
| to a table
|
| * 8.0.13 added DEFAULT expressions (column default values
| may now be any arbitrary expression, not just a constant)
|
| * 8.0.16 added CHECK constraints
|
| * 8.0.17 added the CLONE plugin for rapid physical
| copying of a db
|
| * 8.0.23 added INVISIBLE columns (excluded from SELECT *)
|
| These are just a handful among many others... basically
| MySQL 8 does not follow SemVer at all, nor does it claim
| to. (Although personally as a tool developer in the mysql
| ecosystem, I selfishly wish they did!)
| dafty4 wrote:
| Thanks, HermanL and PradeepN, for sharing this migration story
| with HN!
|
| You wrote:
|
| "Despite all the hurdles in our migration path, we have already
| seen the benefits of running 8.0."
|
| May I ask what are the main performance benefits you have
| measured/noticed?
| theandrewbailey wrote:
| Anyone else notice that bad scaling and jumping on the animation?
| If they're going for an old informational film aesthetic,
| awesome!
| desireco42 wrote:
| So first sentence is:
|
| "MySQL, an open source database developed by Oracle, powers some
| of Facebook's most important workloads. "
|
| Oracle didn't develop MySQL, they bought it... Am I nit-picky
| here, opinions?
|
| Otherwise, congrats on shipping!
| ksec wrote:
| I would have had problems too if this was early 2010s, but I
| mean Oracle has been developing MySQL for more than a decade
| now, so I think it is fair to say "developed by Oracle".
| suyash wrote:
| Go MySQL - solid open source database rocking for the last almost
| 3 decades!
| hodgesrm wrote:
| Well...I love MySQL but there was a time when it was anything
| but rock solid. It was pretty painful until InnoDB replaced
| MyISAM in 2010 as the default storage engine. After that it
| became rare to lose data.
| chovybizzass wrote:
| I wonder if FB could run on sqlite
| dpcx wrote:
| What I take away from this is that they are (or were, until
| recently) still running production systems on MySQL 5.6...
| gregoriol wrote:
| Some migrations are very hard!
| filmgirlcw wrote:
| Yeah, but a highly-customized version of 5.6 with its own
| storage engine. It's not like it was off the shelf MySQL 5.6.
| thebouv wrote:
| Only commenting to say their site makes me mad cause it breaks
| the browser history. I click the link, go to the fb engineering
| site, and the back button has no history to go back to HN. Dark
| Pattern UX -- gross.
| biesnecker wrote:
| Works fine for me in Chrome 91.0.4472.164 on a Mac.
| Semaphor wrote:
| Can't confirm. It works exactly as expected in Firefox (private
| mode because FB links open in the FB container otherwise), so
| maybe it's some extension you have installed? FB Engineering
| links are usually pretty clean, and they don't even have a
| reason to try and keep you there.
| mrcarruthers wrote:
| Yeah if you're using Firefox and you have the FB Container,
| it's working as advertised
| forgotpwd16 wrote:
| Tried with clean profile Firefox and Chromium. Back button
| works fine with both.
| takeda wrote:
| Perhaps it's due to me being in the PostgreSQL camp, but what
| happened to MariaDB? I was convinced that's where MySQL
| developers went and that's where development proceeded, while
| MySQL was just half dead, because Oracle did not care about it.
| unknownOrigin wrote:
| What happened to MariaDB? Not much, but unlike Netbeans (which
| is a complete disaster under Apache... a dying project
| maintained by a skeleton crew), it still has a commercial
| entity behind it, so it's trucking along relatively nicely. I
| mean, it's not like Oracle is investing tremendous amounts of
| time and money into MySQL either... there are still many, many
| things in the MySQL ecosystem that are second class citizens
| when compared to the big boys. Features like materialized views
| are yet to materialize etc.
| Rapzid wrote:
| One thing of extreme value to come out of MariaDB is the JDBC
| driver that explicitly allows you to bundle it wherever you
| want.
| lucideer wrote:
| That was the expectation but the Oracle camp surprised everyone
| by actually giving it some attention, and meanwhile MariaDB's
| promise of being a "100% compatible drop-in replacement" came
| with more and more caveats with each release (as is to be
| entirely expected with any fork).
|
| When you consider the network effect of all the stacks already
| heavily invested in MySQL, all Oracle really needed to do was
| put in a modicum of effort to MySQL to stave off the MariaDB
| migrations (and from my understanding they've put in quite a
| bit more than a modicum).
|
| Lastly, Postgres is an interesting player. Migrating from MySQL
| to Postgres might be a scarier prospect than to MariaDB, but
| since the Oracle acquisition a lot of MySQL-veterans were
| looking for something non-MySQL for _new_ projects and in that
| I get the sense Postgres beat out MariaDB.
| munk-a wrote:
| I carried out a (quite ill-advised in retrospect) migration
| from MySQL to Postgres on a mature codebase with absolutely
| no automated test coverage and it went pretty smoothly. We
| had some bumpy performance directly after the production
| migration but we were able to reach parity after about two
| months of observing and adding targeted indices.
|
| I absolutely adore Postgres's tweaks to the SQL language,
| that dialect is amazing and has served us extremely well.
| And, since we passed pg10 a while back, the performance
| tuning you can do on it is pretty amazing.
| lucideer wrote:
| I am very much an example of one of those previously MySQL
| devs that now uses Postgres for new projects, though I've
| never done a migration between the two.
| code_duck wrote:
| I had an app with a fairly complex and large set of tables
| and data and migrated it from mySQL to pg with very little
| trouble. It was a few years back, but from what I recall a
| re-wrote a few queries in my data layer and that's about
| it. Much easier than it was to setup nginx to take over
| some Apache duties.
| tapoxi wrote:
| We also did a MySQL 5.7 to PG 9.6 migration (thanks to
| https://pgloader.io/) and we got some pretty immediate
| speed improvements (2-5x).
|
| We shot ourselves in the foot a few weeks later with some
| poorly written RLS policies, but it was overall a smooth
| experience.
| twic wrote:
| > I absolutely adore Postgres's tweaks to the SQL language
|
| Do you mean additions to the SQL standard, or bits of the
| standard that MySQL doesn't implement, or something else?
|
| The reason i ask is that i also enjoy writing SQL for
| PostgreSQL, but as far as i know, i am sticking to standard
| SQL. Perhaps there are things i'm missing, or things i like
| which i haven't realised are nonstandard!
| munk-a wrote:
| Personally, back before JSON became a standard thing - I
| made heavy use of HSTORE to clean up some insanely
| cludgey string packing BS we'd been doing - leveraging
| recursive CTEs using the UNION definition allowed us to
| move a fixed tier system into a self-referential one that
| saved us a lot of headaches as well.
|
| Other folks have mentioned :: instead of CAST AS (which
| is really nice when you need to write it a bunch of
| times) but it's mostly the diverse built-in function set
| and constraint support that keeps me an enthusiastic
| Postgres supporter.
| paulddraper wrote:
| > as far as i know, i am sticking to standard SQL
|
| PostgreSQL documentation is very good about comparing
| each feature to the standard.
|
| For example, the SQL standard specifies that triggers
| fire in the historical order they are added to the table.
|
| PostgreSQL chose to fire them alphabetically.
|
| Technically nonconformant, but in practical terms, far
| better.
| eatonphil wrote:
| MySQL does not conform to the standard in a number of
| ways. On the one hand, thankfully you can turn on ANSI
| MODE. On the other hand if you're using MySQL you've
| probably got a bunch of devs used to their non-ANSI SQL
| and turning on ANSI MODE may break stuff they do.
|
| https://dev.mysql.com/doc/refman/5.7/en/sql-
| mode.html#sql-mo...
| jacobmischka wrote:
| I assume they mean non-standard additions in PostgreSQL.
| A lot of the very handy pg functions are non-standard, as
| well as the rich text matching capabilities, many data
| types, and even the RETURNING clause, I believe.
| felixfbecker wrote:
| There is definitely stuff that is non-standard, like the
| casting with :: in addition to CAST AS.
|
| But it's true that PSQL sticks very closely to the
| standard or only adds on top of it, while MySQL is very
| uncomformant.
|
| Funnily enough, I actually first used Postgres over MySQL
| a long time ago when I moved from PHP to NodeJS, and was
| annoyed that MySQL uses backticks for identifiers while
| PSQL uses SQL standard double quotes, because in JS
| multi-line strings (for SQL queries) are also use
| backticks and I was annoyed by escaping. Sometimes it's
| the little things...
| petergeoghegan wrote:
| It is practically essential for any RDBMS to provide non-
| standard statements and features. For example, even
| CREATE INDEX isn't in the SQL standard, since the SQL
| standard deliberately avoids talking about what it
| considers to be implementation details. This is probably
| a good goal in many ways. But what counts as an
| implementation detail is open to interpretation.
|
| You could say the same thing about the isolation levels.
| They are supposed to be platonic ideals, free from any
| implementation baggage. But the reality is that you can
| tell that the people that originally defined how they
| work were mostly (perhaps entirely) thinking about old
| school two-phase locking.
|
| I'm not a critic of the standard -- it's imperfections
| (which are arguably contradictions) reflect real world
| differences that are hard or impossible to resolve. Just
| pointing out that this is how it is.
| munk-a wrote:
| Oh, it's totally fair to criticize the standard. Those
| clowns lock their deliberation and standard definitions
| behind huge paywalls.
|
| The best way to learn the SQL standard is to read the
| Postgres docs since they clearly call out non-compliances
| - there is no other easy to access and clear explanation
| of the SQL standard.
| gunapologist99 wrote:
| If you use any of the really amazing JSON or Array stuff,
| you're stepping outside of the standard.. but it's worth
| it!
| dkersten wrote:
| I absolutely love Postgres' JSON and array support. Its
| saved me from needing a document database in a few cases,
| which drastically simplified the overall deployment (one
| DB instead of two, or trying to represent relational
| stuff in a document DB). I really like JSON for stuff
| that is user defined, that I don't need to query into,
| but postgres' support means I still can should I ever
| need to. Which has happened once or twice for debugging.
| tomxor wrote:
| > meanwhile MariaDB's promise of being a "100% compatible
| drop-in replacement" came with more and more caveats with
| each release
|
| Anecdata: I've been using mariadb as a drop in mysql
| replacement for over 5 years in production, and it's been
| working seamlessly.
|
| I'm no DB expert, and perhaps my use case isn't complex
| enough or written in a highly mysql dependent way - but I
| felt like _someone_ should chime in since there 's not much
| positivity toward mariadb in this thread so far. I'd be
| interested to hear from some other people with _actual_
| mariadb experience.
| sltkr wrote:
| I was curious so I looked up the differences here:
|
| https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility/
|
| It doesn't sound too bad. The incompabilities seem to be
| mostly stuff like not being able to generally use replication
| from MySQL to MariaDB or vice versa. But I would guess that's
| not a very common case anyway: most users are either
| developers that want their software to work with both MariaDB
| and MySQL (which can be achieved by sticking to the very
| large intersection of supported features), or they are users
| that want to switch from MySQL to MariaDB (which seems well-
| supported). Running a mix of MariaDB and MySQL servers and
| expecting to be able to replicate between them seems like a
| particularly uncommon setup (though maybe it's useful during
| a migration).
| hodgesrm wrote:
| Binlog incompatibility makes migration a lot more difficult
| at scale.
| hellcow wrote:
| The issues I encountered while using MariaDB in place of
| MySQL were subtle -- most things worked as expected, which
| makes the issues so much more unexpected. But at the end of
| the day, the same migrations produced a different schema.
|
| Arguably MariaDB made the correct decision to fix some
| MySQL 5.7 bugs that we were unwittingly relying upon, but
| the different behavior caused issues for us all the same.
|
| Rather than worry about MySQL vs MariaDB and whether the
| one I need is packaged and available for my distro/OS of
| choice, I reach for Postgres on new projects.
| PeterZaitsev wrote:
| I think this is a key difference - where MariaDB
| continues to evolve MySQL 5.6/5.7 codebase Oracle chose
| the "big leap" with MySQL 8 - introducing data dictionary
| and many other internal architecture changes. This is one
| of the reasons migration to MySQL 8 can be more painful
| than releases before it but for good reason
| skrebbel wrote:
| Anyone on the in-crowd who can share a bit about Oracle's
| motivations here?
|
| I recall a Bryan Cantrill talk about how Oracle is this
| completely amoral machine that just wants to make money, and
| how that drives their every decision [0]. Is this no longer
| the case? Or is it, and if so, how does improving MySQL and
| giving it away for free making Oracle money? Or was it always
| hyperbole and Oracle is improving MySQL because they want to
| be nice?
|
| [0] https://www.youtube.com/watch?v=-zRN7XLCRhc&t=2047s, that
| link starts at the beginning of what's popularly known as
| "the lawnmower rant", it's delightful.
| takeda wrote:
| I guess I'm cynical, but to me looks like Oracle renewed
| interest in MySQL after PostgreSQL gained momentum. MySQL
| is not really a competitor to Oracle but PostgreSQL
| overlaps with both of them.
| beberlei wrote:
| Oracle is selling Enterprise features and Support, i am
| sure they make a lot of money with MySQL.
| freedomben wrote:
| On Fedora/RHEL you get maria when you "install" mysql. I've
| never had something fail to "just work." That said I don't do
| anything crazy, so I'm sure there are cases where they don't
| work.
| z3t4 wrote:
| When I started working with databases about 20 years ago
| talking to peers it was common to have a layer in-front of
| the database so that you could easily replace the database.
| It felt stupid until I actually had to do a database
| migration... So do you still put a layer infront of the
| database so that you can for example easily change from mySQL
| to Postgress? Given there are so many open source database I
| think there is less need to do so... Or do you go all in
| vendor lock in? Or some middle ground?
| astrange wrote:
| That would be an ORM or something similar, I think. It's
| not realistic unless you're always testing with two
| databases from the start, because their implementation
| details leak through.
|
| It can be useful if your program is simple enough to work
| with both MySQL and SQLite, but often you should just stick
| with one DB and get to use all its features.
| pyrophane wrote:
| I like Postgres but still turn to MySQL for production use
| most of the time for a few reasons:
|
| * Permission management on Postgres is much more painful. *
| The need for an external connection pooler makes postgres
| more annoying to set up. * Better quality docs on performance
| tuning MySQL.
| agumonkey wrote:
| sounds like their java strategy until recently
| PeterZaitsev wrote:
| I do not think there was the time when MySQL was quite dead
| under Oracle ownership. As far as I aware Engineering team in
| particularly have only been steadily growing.
|
| It is fair to say Oracle did not invest in ease of
| Oracle->MySQL migration but they have been making a steady
| progress on making MySQL better database for modern
| applications
| MangoCoffee wrote:
| MariaDB is getting more corporate. it feel like Michael
| Widenius's MySql second act.
___________________________________________________________________
(page generated 2021-07-22 23:00 UTC)