[HN Gopher] Migrating Facebook to MySQL 8.0
___________________________________________________________________
Migrating Facebook to MySQL 8.0
Author : moneil971
Score : 490 points
Date : 2021-07-22 17:54 UTC (1 days ago)
(HTM) web link (engineering.fb.com)
(TXT) w3m dump (engineering.fb.com)
| wewe2fg23 wrote:
| Only oldies uses Facebook in 2021
| andremendes wrote:
| Anyone else puzzled by the shaking migration animation? The
| original file has a 1920x1080 resolution. I wonder if it is
| related to the shaking. Good article nevertheless.
| andrewmcwatters wrote:
| If you look closely, you can see the engineers nervously
| rolling out changes to production, which is where the
| vibrations in the .gif come from.
| Normal_gaussian wrote:
| It has ?w=1024 causing resizing effects:
|
| Shaking: https://engineering.fb.com/wp-
| content/uploads/2021/07/CD21_3...
|
| Not shaking: https://engineering.fb.com/wp-
| content/uploads/2021/07/CD21_3...
| zestyping wrote:
| I see it too. It's weird and distracting -- something must have
| gone wrong in the rendering.
| pradeep1288 wrote:
| thanks, this is fixed now :)
| andrewmcwatters wrote:
| I wonder what order of magnitude of rows they deal with and how
| they optimize indices for querying that data.
| datavirtue wrote:
| Like everyone else?
| andrewmcwatters wrote:
| I didn't downvote you, but you were downvoted for this. So
| maybe I'll explain further, instead, to stir conversation.
|
| As you deal with more and more rows, it becomes imperative
| that all of your where clauses hit indexes/indices, but even
| beyond that, with large enough row sizes, it's not enough to
| provide fast responses.
|
| I suspect they deal with this in the form of some sort of
| caching outside of MySQL, but I haven't read into it.
|
| I'd be curious about how you respond to such challenges at
| the billions or trillions of rows orders of magnitude.
| Millions can be difficult enough, but beyond that you may
| have queries that effectively never return if you do not
| plan.
|
| Related reading:
|
| [1]: https://dba.stackexchange.com/questions/20335/can-mysql-
| reas...
|
| [2]: http://webscalesql.org
| evanelias wrote:
| Facebook is heavily sharded, which keeps the size of each
| physical table at reasonable levels.
|
| Similar story at nearly every large tech company using
| MySQL, aside from some more recent ones that go for the
| painful "shove everything in a huge singular AWS Aurora
| instance" approach :)
| andrewmcwatters wrote:
| Thanks for that insight! Sounds like maybe you've seen an
| interesting thing or two given your background.
| vosper wrote:
| > the painful "shove everything in a huge singular AWS
| Aurora instance" approach :)
|
| I haven't used Aurora - what's painful about this
| approach (apart from potentially being locked-in to AWS /
| Aurora)?
| evanelias wrote:
| Aurora lets you grow your db storage up to 128 TB, but
| things become difficult and slow long before this point.
| This is true operationally (e.g. schema changes,
| restoring from a backup, etc) as well as potentially for
| application / query performance issues.
|
| That said, sharding can be very painful too in other
| ways. But at least it permits infinite horizontal
| scaling, and it makes many operational tasks easier since
| they can be performed more granularly on each smaller
| shard.
|
| Besides, once you get anywhere near Aurora's storage
| limit, you'll need to shard _anyway_. So it really just
| buys you some time before the inevitable.
| vosper wrote:
| Interesting, thanks for the insights
| ahiknsr wrote:
| > I suspect they deal with this in the form of some sort of
| caching outside of MySQL, but I haven't read into it.
|
| It is handled by TAO,
| https://engineering.fb.com/2013/06/25/core-data/tao-the-
| powe...
| andrewmcwatters wrote:
| Thank you for sharing this!
| thayne wrote:
| > A few applications hit repeatable-read transaction deadlocks
| involving insert ... on duplicate key queries on InnoDB. 5.6 had
| a bug which was corrected in 8.0, but the fix increased the
| likelihood of transaction deadlocks.
|
| Interesting, I've run into a bug in MySQL 5.6 involving select
| for update where it deadlocked when it shouldn't. I wonder if the
| two were related.
| fukmbas wrote:
| Lmao FB is on mySQL? Someone fucked up
| markus_zhang wrote:
| >It took a couple of years to complete porting all of these
| features. By the time we got to the end, we had evaluated more
| than 2,300 patches and ported 1,500 of those to 8.0.
|
| This is some tremendous effort as I see. I wonder how many people
| work on these evaluations.
| 1024core wrote:
| > MySQL, an open source database developed by Oracle
|
| This makes it sound like Oracle created it, when it fact the
| reality is the opposite: Oracle fought it, and bought Sun just so
| they could get their hands on MySQL. How did regulators let that
| happen is beyond me.
| jahewson wrote:
| No. The most valuable part of sun at the time was arguably the
| SPARC chip architecture and Solaris which gave Oracle a
| vertical compute platform to compete with IBM Db2 on POWER
| running AIX. In our new cloud world this is now quaint and
| irrelevant but it was a big deal when on-prem was the only way.
|
| There was also Java.
| cat199 wrote:
| sun had plenty of valuable assets in addition to mysql - don't
| think one can reasonably claim they bought sun _just_ for mysql
| without inside knowledge
| hitekker wrote:
| Seems like good, old-fashioned engineering work to me. No
| breathless evangelism or careerist claptrap in sight.
| linspace wrote:
| It seems that "move fast and break things" is not for billion
| dollar companies.
|
| My impression too. Very meticulous work, just the facts.
| gigatexal wrote:
| Where can I read more about the custom things FB added to MySQL
| 5.6? Their work on replication and the DDL work seems
| fascinating. Are there scholarly papers, white-papers, blogs?
|
| Are they storing the friend graph in MySQL?
| midom wrote:
| there's also this team page, although it has been somewhat
| lower on volume lately -
| https://www.facebook.com/MySQLatFacebook/
| gigatexal wrote:
| I've not got a FB account due to reservations about FB and
| their business model but that doesn't mean I am above
| learning from what they do, thanks for the link, I'll have to
| find info elsewhere though.
| throwdbaaway wrote:
| I don't think you need a FB account to read the posts
| there, you can just treat it like a blog with a crappy
| interface (but very good content). Probably some mandate to
| "eat your own dog food".
| HarrisonFisk wrote:
| Direct source is available at:
|
| https://github.com/facebook/mysql-5.6
|
| The social graph is indeed stored in MySQL via TAO. A recent
| white paper about MyRocks is:
|
| https://research.fb.com/publications/myrocks-lsm-tree-databa...
| ablekh wrote:
| > MySQL, an open source database _developed_ by Oracle
|
| Dear Facebook:
|
| Please fix the wording emphasized above ASAP before we revolt
| here in outrage with your lack of attention to detail. MySQL was
| developed by relevant open source project contributors
| (initially, MySQL AB folks), but not by Oracle and even not by
| Sun Microsystems. MySQL has been in development for ~12 years
| before Sun acquired MySQL AB. Please set the record straight.
|
| Sincerely,
|
| MySQL user (not currently, but in the past and maybe in the
| future [though unlikely, because Postgre...] :-)
| throwdbaaway wrote:
| > First, we could not upgrade servers in place and needed to use
| logical dump and restore to build a new server. However, for very
| large mysqld instances, this can take many days on a live
| production server and this fragile process will likely be
| interrupted before it can complete.
|
| This part is quite interesting. I'd think it will take maybe a
| day to dump/restore a database with a single 10TB table with no
| parallelism, at a speed of about 120MB/s. How big is these mysqld
| instances such that the process will take many days?
| alberth wrote:
| It's interesting to see FB migrating so much over to Oracle.
|
| 2 weeks ago was FB use of GraalVM (from Oracle). And now this.
|
| https://news.ycombinator.com/item?id=27782475
| dcolkitt wrote:
| I guess one could dream of the remote possibility that Facebook
| outright acquires Oracle. It seems like a weird match, but it'd
| give FB a toehold in a lot of verticals that it currently has
| no presence in. Whatever criticisms you have of FB, it's at
| least a pretty open source friendly company.
| amelius wrote:
| What would happen to Oracle's infamous legal department then?
| munk-a wrote:
| Then they'd leverage social media oversharing to find more
| targets to litigate - it'd be a win-win! /s
| aloisdg wrote:
| You mean pretty much the whole company? /s
| rattray wrote:
| Fabulously lavish retirements?
| dylan604 wrote:
| They'll start serving ads and sharing personal data of all
| of the people they sue?
| rrdharan wrote:
| Facebook has no interest in infrastructure / enterprise
| revenue streams (Facebook for Work notwithstanding). See the
| history of Asana, Cloudera and Phabricator (and even Dropbox,
| from an employee/recruiting perspective, kinda).
| hodgesrm wrote:
| The MySQL 8.0 upgrade is not a migration to Oracle. Facebook
| has run MySQL for many years at this point and, as the article
| describes, operates on a private fork. They have employed some
| of the leading engineers in the MySQL community to support
| their version of MySQL.
| maxk42 wrote:
| Oracle has a very complete and much cheaper cloud solution
| relative to AWS. I've begun migrating my own projects over to
| their cloud. The savings on egress costs alone pays dividends.
| mr_overalls wrote:
| I find the comments in this article very accurate:
|
| https://news.ycombinator.com/item?id=15160149
|
| > ORA is the elephant's graveyard of software.
|
| > Once something gets bought by them, you know it is done.
| Slowly, but surely.
|
| > They perform a function akin to the maggots that destroy
| cadavers in nature. Part of the overall ecosystem.
|
| > ORA stopped being a tech co a while ago, now it is a
| finance play. Use cash to buy a business for its locked in
| customers, gut it to squeeze max money out of it until last
| customer is gone. Rinse, repeat.
| adrianmsmith wrote:
| This is not so with Java. They have moved Java along a lot
| since they bought it; they released 1.8 finally, and have
| introduced lots of new features such as type inference
| since then.
|
| MySQL is also going strong after the acquisition. Lots of
| new features.
| BossingAround wrote:
| And yet MySQL is very much alive..? Java has fully
| opensource implementations too, and has had innovations
| like GraalVM and Jakarta EE.
|
| I think I'm too young to hate Oracle with a burning
| passion. I would be very scared of their sales/corporate
| team, but I mean, even having large contracts with Google
| or Amazon is difficult to say the least.
| danpalmer wrote:
| MySQL is alive for companies already using MySQL, but
| would well informed companies be choosing it today? I'm
| not so sure.
|
| I think you either want it to have more functionality and
| you choose Postgres, you want a particular performance
| profile and you pick MariaDB, or you want the enterprise
| support and you pick Oracle's database. Why you would
| pick Oracle-supported MySQL today I'm not sure.
| abledon wrote:
| didn't the new basecamp "Hey" email product choose to use
| MySQL?
| danpalmer wrote:
| I think so, but they have been operating MySQL at scale
| for many products for about 20 years, so I can understand
| why they might not choose MariaDB (or Postgres, Oracle,
| etc).
| ericyan wrote:
| Easy to operate, perhaps?
|
| With MySQL group replication, it is pretty easy to set up
| a HA cluster that allows you to sleep soundly. It can
| also handle thousands of connections without the need of
| a connection pooler.
| danpalmer wrote:
| Sure, but MariaDB is the more obvious choice surely?
| evanelias wrote:
| No, why do you think so?
|
| Among the largest users, MySQL (or Percona Server) is
| chosen far more frequently than MariaDB.
|
| GP mentioned Group Replication, which isn't even
| available in MariaDB.
| threeseed wrote:
| > have more functionality and you choose Postgres
|
| PostgreSQL still after all these years has a terrible
| story when it comes to HA and horizontal scalability.
| Nothing is built in or supported. And one of the only
| companies to provide something half decent i.e. Citus is
| now owned by Microsoft and so that project is now at
| risk.
|
| At least MySQL has something built-in.
| danpalmer wrote:
| So MariaDB then.
| kaba0 wrote:
| That one is more on MySQL, isn't it? They were the ones
| that sort of stopped innovating for a time.
| redis_mlc wrote:
| > I think you either want it to have more functionality
| and you choose Postgres
|
| False. MySQL and Postgres are comparable feature-wise.
|
| @danpalmer: you're making an ass of yourself. It's clear
| from your comments that you're around 10 years out of
| date.
| VintageCool wrote:
| Most people seem to be using the Percona Server branch of
| MySQL.
| danpalmer wrote:
| Good point, again this seems like another alternative
| that makes "stock" MySQL an option that there seems to be
| no good reason to choose.
| kaba0 wrote:
| That's the usual blind rhetoric against Oracle. Any factual
| info on how are they worse than the "good" companies that
| literally track every single step you take, that attack the
| very democracy of countries, that help the spread of
| dangerous anti-intellectual rhetorics, or the "usual" anti-
| union, shitty working condition giants?
|
| They dared sue the small mom&pop shop called Google for
| going against the Sun license of Java?
| alberth wrote:
| Just curious, so which companies do people consider are
| creating/selling interesting and innovative ENTERPRISE
| software?
| llimos wrote:
| Microsoft. King of the enterprise with no intention of
| losing the title.
|
| PowerApps, Power Automate etc. - while not necessarily
| innovative in a vacuum - are, when you consider they are
| bringing cloud-based automation and RPC to the
| (enterprise) masses.
| cutthegrass2 wrote:
| HashiCorp.
| voidfunc wrote:
| This is such a hilariously myopic HN take.
| mr_overalls wrote:
| I work for a government agency in a rather poor flyover
| state. We ran Oracle databases and a few other products
| for multiple decades, until Oracle conducted an audit and
| decided to shake us down for everything we were worth. We
| migrated away from their products, but only after being
| forced into expensive multi-year subscriptions for
| products that we didn't need.
|
| I will never forget the arrogance and greed of their
| legal/sales people. They are bloodsucking sociopaths.
| abraae wrote:
| I'm no fan of oracle's audit approach but it sounds like
| you were expecting special leniency to not pay for their
| stuff - for decades - because you are a govt agency/ live
| in a low income place. Not how business works. If you're
| playing by the rules, no one can force you into any long
| term contracts you don't need.
| myohmy wrote:
| I'm also a government Oracle dev. I have no way of
| knowing what I'm licensed for and they don't disable the
| features we're not licensed for. I literally have to talk
| to my director to talk to our account rep to figure out
| that I'm not allowed to use table partitioning.
| chasd00 wrote:
| portions of Salesforce are the same way. You don't know
| you're out of compliance until an audit happens and you
| get the "sure would be a shame if something happened to
| your instance..." speech.
| abraae wrote:
| Agreed that Oracle licensing is byzantine, probably
| deliberately. But devil's advocate says that any
| organization should be aware of what they are actually
| licensed for, even if that means contracting a third
| party to ensure compliance.
| mr_overalls wrote:
| We have a small IT department and are running on
| extremely limited funds. We literally had two unlicensed
| dev databases, and Oracle threatened to sue us for
| millions of dollars, unless we bought a subscription to
| their shitty cloud product.
|
| The whole process was shitty, predatory, and abusive.
| sly010 wrote:
| Totally off topic, but I found it funny that in the offical
| oracle cloud intro video the robot arm in the background
| misses the box, right as the narrator mentions the advantages
| of automation at 1:10:
| https://www.youtube.com/watch?v=f_BDfMdGn3M&t=61s
| suyash wrote:
| lol good eye! but it shows me how real tech works, nothing
| is 100% guranteed
| wizzwizz4 wrote:
| I think you'll find the box missed the robot arm.
| abledon wrote:
| I hate oracle, but man do i love their art style branding,
| kind of a 'painting' vibe
| topspin wrote:
| > Oracle has a very complete and much cheaper cloud solution
| relative to AWS
|
| Tell us more. Ignore the haters please. I'd love to have your
| insights.
| maxk42 wrote:
| It's all on their site - https://www.oracle.com/cloud/
| Their price list is at https://www.oracle.com/cloud/price-
| list.html and the egress is under the "networking" section.
| Basically they give you 10TB outbound data transfer for
| free and charge as little as $0.0085 per GB after that.
| That's a savings of 90% (or 100% if you use less than 10TB
| transfer, which most apps do)
|
| Additionally their cloud servers (and most other services
| as well) are much cheaper than comparble EC2 instances and
| every one of their services I've used so far also includes
| a free tier so you can try them out without paying
| anything.
|
| All in all pretty pleased so far.
| gunapologist99 wrote:
| Yeah, AWS (and Azure, and GCP) really has notoriously
| expensive bandwidth, and it's the one thing they will
| never budge on in negotiations, since they know it's
| where they have nearly 100% profit margin; you can avoid
| using most services, or use compute etc more efficiently,
| but it's hard to consume less bandwidth!
|
| It sounds like Oracle is actually pricing bandwidth at a
| reasonable price. This is quite a shock, coming from a
| historical perspective!
| luhn wrote:
| That has nothing to do with MySQL. Facebook operates their
| own DCs and hasn't indicated any interest migrating
| elsewhere.
| jmnicolas wrote:
| Tying yourself to Oracle is quite a risky choice in my
| opinion.
|
| After having used their non cloud products 10 years ago, I
| wouldn't do any business with them. At all.
| na85 wrote:
| I'd say the same about Facebook.
| suyash wrote:
| You can see Oracle or you can see Open Source Awesome
| technology they are using, both of these are solid OSS
| projects.
| cranekam wrote:
| It's a stretch to claim this is a "migration". Facebook has
| used MySQL since its very early days -- certainly since before
| Oracle acquired Sun and thus MySQL AB.
| slownews45 wrote:
| Somehow Facebook and Oracle seem like fitting partners. Too
| horrible companies. One fighting the fight with lawyer / sales
| teams, the other with clickbait / outrage
| sdevonoes wrote:
| Do you know if they use the Percona toolkit with MySQL? Does it
| even give any benefits if one is running MySQL 8?
| simonw wrote:
| I'd love to know roughly how many engineers (or engineer hours)
| this took - sounds like a mammoth project!
| schuyler2d wrote:
| I'd be interested more in what features were in their (3):
|
| > Non-MyRocks Server: Features in the mysqld server that were not
| related to our MyRocks storage engine were ported.
|
| i.e. things that they haven't shared with the MySQL community but
| aren't related to the storage layer. Why would/should they keep
| those private? Considering that RocksDB is open-source, clearly
| it's unlikely it's because they're 'trade secrets'.
| HarrisonFisk wrote:
| The code is open source:
|
| https://github.com/facebook/mysql-5.6
| sriram_sun wrote:
| "MySQL, an open source database developed by Oracle" - Gawd! Were
| most devs at FB born after the acquisition?
| evanelias wrote:
| It didn't say _originally_ developed by Oracle. The statement
| is accurate; MySQL is currently developed by Oracle.
|
| At big companies, sometimes the corporate lawyers are overly
| picky about wording in blog posts and public presentations,
| e.g. "make it clearer that MySQL isn't our in-house product"
| type of thing.
|
| fwiw, several of FB's top database folks originally worked for
| MySQL AB, so they're well aware of the development history.
| cmrdporcupine wrote:
| My old-man brain still can't compute this statement: _" MySQL, an
| open source database developed by Oracle"_. I know this is now
| factually true, but late-90s me keeps looking over my shoulder
| and freaking out.
| soperj wrote:
| should read: "an open source database bought by Oracle".
| kaba0 wrote:
| And then further developed...
| blantonl wrote:
| The hair on my neck still stands up when thinking about dealing
| with Oracle (and Sun) contracting, purchasing, and sales teams
| in the 90's.
| briffle wrote:
| I dealt with them in 2016. I assure you, not much has
| changed.
| cmrdporcupine wrote:
| Don't forget IBM
| beefield wrote:
| I have managed to find myself accidentally twice at the
| receiving end of IBM "services". Conclusion was pretty much
| same both times:
|
| Never. F###ing. Again.
| rwallace wrote:
| Traditionally, part of what made it worth paying IBM
| prices was reliably top-notch service, but perhaps that
| changed over time. What were your experiences? And which
| decade?
| silicon2401 wrote:
| What's it like working at IBM? I've seen it get good
| ratings online
| forbiddenlake wrote:
| This question is extremely hard to answer, because IBM
| has ~345k employees in 177 counties. It's anywhere from
| 0/10 to 10/10.
| asdfasdfef wrote:
| * -10/10 and 0/10
| zerop wrote:
| Agree. I have used IBM web sphere in 2008. IBM, Oracle, Sun
| is a tribe.
| filmgirlcw wrote:
| I had the same reaction! I was like, "technically, yes, but..."
|
| I haven't thought about MySQL proper in years, but it is
| interesting Facebook is moving to MySQL 8 rather than going to
| MariaDB, which as far as I know, had MyRocks built-in.
| dheera wrote:
| I just configured a new router and had to set up my dynamic DNS
| updating service on it.
|
| Went to dyn.com and the goddamn website is gone, redirects to
| some corporate Oracle bullshit and PR, and I can't login to
| that Oracle crap with my dyn.com account and password.
|
| In case any confused people googling for this stumble upon this
| comment you need to go to: account.dyn.com which they don't
| tell you on the Oracle page.
|
| You can log in there with your former dyn.com credentials.
| fredliu wrote:
| I can't read pass "developed by Oracle". It is factually not
| true. Unless there's so much code change after the acquisition
| that there's no single trace of the original open source
| version of MySQL left in there.
| johannes1234321 wrote:
| MySQL always was primarily developed by an commercial entity.
| And always gave an license out under GPL (i believe very
| first used some other license as GPL wasn't known back then
| ...) Oracle took over the development team (lots of pre-
| acquisition staff included) and legal rights to all of that.
|
| (Disclaimer: I started at independent MySQL AB more than ten
| years ago and now wear an Oracle badge, after wearing Sun for
| a while)
| fredliu wrote:
| Maybe it's legal talk that I don't understand, but the fact
| that Oracle owns the majority of the dev team _today_ ,
| doesn't mean Oracle developed MySQL. Because at the time
| MySQL was "developed", Oracle didn't own MySQL, nor did
| Oracle "own" the developers.
| johannes1234321 wrote:
| Oracle still develops and to a notable number the same
| people. Also Oracle steered 8.0 which was a big change to
| internals. Anyways won't argue about single words.
| quesera wrote:
| "Developed" is not used in the past tense here.
|
| Think " _Currently_ developed by Oracle ".
| PeterZaitsev wrote:
| I think MySQL Engineering team does great job with Engineering,
| Do not want to deal with Oracle Sales ? I would invite you to
| talk to Percona for Commercial Support and Services Options
|
| Full disclosure - I'm CEO at Percona :)
| dvtrn wrote:
| _Full disclosure - I 'm CEO at Percona :)_
|
| oh hi. big fan of you all. xtrabackup and PMM are wonderful
| tools
| PeterZaitsev wrote:
| Thank you!
| zepearl wrote:
| I got sad when I read that TokuDB in Percona & MariaDB was
| being dropped :(
|
| Replaced TokuDB with MyRocks but it's totally different =>
| now finally managed to have good ~constant performance
| insert&deletes (had to test & tune a lot), but "updates"
| remain a problem (they write a lot as I understand that any
| update always has to rewrite the entire row).
|
| Using InnoDb instead of MyRocks is NOK for me (problems with
| deletes and related hole punching + bad that tables can never
| shrink when stuff gets deleted), I don't see other
| alternatives... .
|
| I'm now experimenting with other DBs (right now with
| CockroachDB but it doesn't seem to be very stable, might then
| try as well Postgres but I'm scared by the lack of SQL-hints
| to cover worst case scenarios).
| tpetry wrote:
| You can enforce how to execute parts of your PostgreSQL
| with pg_hintplan. And it has more options than mysql's
| approach ;)
| zepearl wrote:
| Thanks a lot!!! I really mean it!
|
| This might change everything for me. I'm already using
| MariaDB+MyRocks and Clickhouse for some special/dedicated
| tasks, but I was really missing a good DB for
| normal/typical OLTP tasks.
|
| So far I used (as mentioned above) MariaDb+TokuDB (but
| the optimizer of MariaDB can be a bit crazy from time to
| time), had multiple times during the past months thoughts
| about PostgreSQL but the lack of hints always made me
| take a step back from it => this addon seems to be
| exactly what I wished for.
|
| Looks like that my weekend will be all about PG - the
| last time that I set it up its version number had a
| single digit => cannot remember anything anymore... .
|
| Again, thanks for the hint :P
| jtbayly wrote:
| Agreed. And in one sense (English is complicated) it's not true
| in its most obvious interpretation--that Oracle is the reason
| it exists. It was developed by others. Changing it to
| "currently being developed by Oracle." It's not that "developed
| by" can't mean "currently being developed by." It just isn't
| the most obvious meaning.
| pmarreck wrote:
| can it do DDL commands in a transaction yet?
| [deleted]
| jjeaff wrote:
| I always see people asking about this. But why? Are you making
| schema changes that frequently?
| handrous wrote:
| Schema changes are pretty much the riskiest damn thing you
| can do with a database, so it's natural people would be
| _extremely_ interested in being able to confine as much of
| that as possible to transactions, even if changes are rare
| (which, arguably, they 're rare in no small part _because_
| they 're so risky).
| simonw wrote:
| It's my opinion that schema changes should be cheap, fast and
| unexciting.
|
| Unfortunately, in my experience once you are operating at
| scale they are expensive, slow and fraught with peril.
|
| The result is that engineering teams often chose to build
| poorly designed schemas and take on technical debt rather
| than make a schema change to a core table. This is a terrible
| anti-pattern!
|
| I'm massively in favour of anything that makes schema changes
| cheaper and more boring. I really enjoy working with the
| Django schema migration system for this reason - though it
| won't help you as much if you need to make schema changes to
| older versions of MySQL (I've not tried schema changes
| against 8.0 yet) since you'll likely still need to use GitHub
| ghost or Percona pt-online-schema-change.
| gfody wrote:
| yeah treating every schema change as a db migration might
| be our industry's worst anti-pattern. transactional ddl
| allows you to do stuff like: begin tran
| ..refactor data, drop/create tables.. ..create
| backwards-compatible views.. commit
|
| ..against a live database without risking consistency
| lclarkmichalek wrote:
| https://github.com/facebookincubator/OnlineSchemaChange
| iamaperson wrote:
| https://www.planetscale.com/blog/non-blocking-schema-
| changes
|
| disclaimer: I work at PlanetScale
| spullara wrote:
| Before Twitter migrated off their MySQL user database there
| were many columns that had been repurposed for exactly this
| reason.
| simonw wrote:
| I've been contemplating adding a JSON column to all of my
| tables just to provide somewhere that covers those cases
| - still controlled with source-code level validation and
| with the expectation that any column which needs to be
| used for filtering rows gets promoted to "real column"
| status.
|
| I'd much rather use a robust, performant real schema
| migrations tool though.
| yxhuvud wrote:
| Yes? Also, I want them to not be able to break halfway.
| hans_castorp wrote:
| > But why? Are you making schema changes that frequently?
|
| Because it makes the migrations much more robust and way
| easier to develop & test.
|
| If something goes wrong in a migration script, everything is
| rolled back. You edit the part that caused the error and
| simply run the whole script again.
|
| And despite of thorough testing, there is always the
| possibility that it fails when applied in production. You
| want that migration to be completely successful or not at
| all.
|
| Think e.g. splitting a table into two (which also includes
| DML) or moving columns around while creating new foreign
| keys.
|
| You might not need to put the whole migration into a single
| transaction, but at least combine some steps into one - but
| this is only possible if the DDL statements are
| transactional.
| pauldino wrote:
| No, but MySQL 8 does at least have atomic DDL now, as well as
| instant DDL for some cases (like adding a column).
| adrianmsmith wrote:
| In terms of more fundamental features: can it do CHECK
| constraints yet?
| mrcarruthers wrote:
| If I remember correctly, 8.0 has them finally. 8.0 seems to
| have fixed a lot of the weird legacy eccentricities of MySQL
| hans_castorp wrote:
| 8.0.16 to be precise
| mrcarruthers wrote:
| Reading the article, seems like facebook has added support for
| it in one of their internal patches.
| spullara wrote:
| Nope. In terms of broad usage I think only Postgres supports
| it.
| gfody wrote:
| sql server and db2 also support ddl in transactions, oracle
| still does not
| SPBS wrote:
| Even SQLite supports transactional DDL. MySQL (and Oracle)
| are the exceptions here.
| hans_castorp wrote:
| Firebird supports it as well
| mackman wrote:
| Wow. I worked on the upgrade from 5.0 to 5.1 (I think). We had
| fewer than 100 patches at the time. 1700 sounds like a nightmare.
| I apologize for any and all of the patches I wrote.
| FpUser wrote:
| >"MySQL, an open source database developed by Oracle, "
|
| Sorry but there is a difference between "developed" as in
| originally created and "being developed" which is the case here
| in relation to Oracle.
| bratao wrote:
| +1 for MyRocks. If you can, try it. We store some text data and
| the compression is over 10 times better than InnoDB with a
| significative speedup.
| ksec wrote:
| MySQL 5.6 was released on 5 February _2013_.
|
| MySQL 8.0 was released on 19 April _2018_.
|
| >The 8.0 migration has taken a _few years_ so far. We have
| converted many of our InnoDB replica sets to running entirely on
| 8.0.
|
| At the scale of Facebook I wonder if they are the largest MySQL
| user on the planet.
|
| And I take this opportunity to ask, does anyone know how does the
| MySQL roadmap works? What sort of features are coming or when is
| 9.0 expected to arrive.
| Thaxll wrote:
| Youtube also use MySQL.
| dheera wrote:
| Wow. Why don't they use in-house Google databases, which are
| supposedly vastly more scalable than MySQL?
| SQueeeeeL wrote:
| Probably because a lot of those benefits don't actually
| translate that well towards their problem space. These
| aren't a bunch of 20 year olds throwing together a MongoDB,
| they're professionals using tools to actually make a
| product
| williamdclt wrote:
| I don't think the people behind Spanner are a bunch of
| 20yo throwing together a MongoDB
| jacquesm wrote:
| Because there is a substantial cost to such migrations. If
| it works don't fix it.
| mastersummoner wrote:
| While "if it ain't broke, don't fix it" is often a
| perfectly valid approach, there are going to be times
| when an upgrade can result in significant performance
| improvements or fewer admin headaches.
| linspace wrote:
| Does anything "just work"? Everything is constantly
| changing, new features always in demand, more users, more
| data.
| lazide wrote:
| And you need to support that for anything you have ON TOP
| of any migration work - which is huge, and also a
| similarly moving target
| myohmy wrote:
| If IT is on the value chain this seems insane. If IT is
| ancillary to the value chain then this is standard.
| vp8989 wrote:
| YouTube built Vitess, which lets you scale out MySQL
| writers. If I can scale out both my writers and my readers
| then what could be "vastly more scalable" than that?
| WJW wrote:
| As far as I can tell, Vitess still has trouble with
| cross-server transactions and distributed locking.
| Presumably something "vastly more scalable" could work
| around this.
|
| The Spanner team noticeably put out
| https://cloud.google.com/blog/products/databases/inside-
| clou... a few years back, which tl;drs that they managed
| to get a reliable enough network that partitions don't
| really happen anymore and so you can get both C _and_ A
| from the CAP theorem. I don't think that Vitess is quite
| at that level, especially since you can self-host it and
| thus the devs don't have such network-level uptime
| guarantees.
| vp8989 wrote:
| Are cross-server transactions scalable? Would anyone
| architect a system that did them en masse unless they
| absolutely 100% had to?
|
| I am just speculating here but I doubt that YouTube
| migrated to Spanner so that they could do cross-server
| transactions. I would think consolidating the Ops burden
| into the GCP org and also serving as a trophy "customer"
| were higher on the list of reasons. But again, just
| speculating...
| derekperkins wrote:
| Yeah, Google policy was to migrate everything to Spanner,
| not a technical decision.
|
| > Are cross-server transactions scalable? Would anyone
| architect a system that did them en masse unless they
| absolutely 100% had to?
|
| As a long time Vitess user and contributor/maintainer,
| this perfectly describes the issue. Minimal effort has
| been put into solving cross shard / 2PC transactions
| because a well designed system rarely needs it.
| ksec wrote:
| I guess people from PlanetScale will know. Not sure if
| they are reading the thread.
| ansible wrote:
| Spanner paper:
|
| https://static.googleusercontent.com/media/research.googl
| e.c...
|
| It is designed from the ground-up to be globally
| distributed. Some other databases like FoundationDB (now
| owned and killed? by Apple) and CockroachDB were designed
| to scale across multiple datacenters.
|
| _Correction:_ FoundationDB is alive as open-source
| again, yay!
| svieira wrote:
| FoundationDB was re-released as open source a couple of
| years ago.
|
| https://www.foundationdb.org/
| imoverclocked wrote:
| AFAIK, FoundationDB is alive and well. What makes you
| think it's killed?
| capableweb wrote:
| It was acquired by a huge company that sometimes kills
| products. Shortly after the acquisition, the downloads
| were removed from the website. It did go open source
| "recently" though which made it longer lasting, but I'd
| understand if someone would think it was dead by now.
|
| - https://news.ycombinator.com/item?id=9259986 - Apple
| Acquires FoundationDB - March 24, 2015
|
| - Comparison between March 25 and March 14, 2015 - https:
| //web.archive.org/web/20150314231702/https://foundatio...
| (14th March) > https://web.archive.org/web/20150325003252
| /https://foundatio... (25th March)
| gresrun wrote:
| YouTube _used_ to use a MySQL-based system called Vitess[0].
| It no longer uses it and has migrated to Spanner.
|
| [0]: https://vitess.io/
| annexrichmond wrote:
| was that migration a technical decision or a political one?
| haimez wrote:
| This is one of those scenarios where the answer is likely
| to be "both", but I doubt you'll get anyone discussing
| YouTube internal architecture in enough detail to be
| satisfying. Just look up what Vitesse is and compare what
| that implies in terms of operational complexity to using
| something like spanner if you want a straw man technical
| rationale.
| Thaxll wrote:
| Thanks for the update! I found more details there:
| https://opensource.google/projects/vitess
|
| > How Google uses Vitess
|
| > Vitess was serving all YouTube database traffic from 2011
| to 2019.
| wikibob wrote:
| Ah this is interesting. Do you have any references on their
| migration to Spanner?
| buro9 wrote:
| gresrun works at YouTube, if by reference you mean
| citation then his comment is it.
| jjnoakes wrote:
| wikibob was probably looking more for public
| documentation about the why and how rather than
| confirmation that the statement is true.
| wikibob wrote:
| Yep, I love watching conference talks or reading Twitter
| threads, papers, etc about these kind of migrations.
| owaislone wrote:
| Probably because Youtube was using MySQL before being
| acquired and switched to a better, probably fully managed
| DB that was much easier (free?) to maintain and scale.
| [deleted]
| dekhn wrote:
| google internally has always had a split mind about mysql. It
| never really jibed with the underlying google architecture
| but played an important role in multiple products (at least
| partly due to its replication features). Replacing it
| completely has turned out to be a huge challenge. I gained a
| fair amount of respect for mysql after seeing it applied at
| scale, but it was also a bit scary.
| merlinscholz wrote:
| The NSA also uses MySQL for their project XKEYSTORE (digital
| surveillance)
| MuffinFlavored wrote:
| > At the scale of Facebook I wonder if they are the largest
| MySQL user on the planet
|
| I have to imagine they use other databases as well, right?
| We're much smaller than Facebook and we have some Mongo and
| some Postgres and a mix of Go/Rust/Scala/node.js.
|
| I wonder how many boxes they have dedicated to be being MySQL
| machines. I wonder how big their tables are and how many writes
| per second they see during the day.
| hodgesrm wrote:
| Their scale is enormous. I first heard the phrase "thundering
| herds" in a Facebook MySQL talk. (Thundering herds refers to
| what happens to your infra when something big happens in real
| life that lights up social networking/search/etc.)
| MuffinFlavored wrote:
| Let's try to put a guess in numbers.
|
| > With roughly 2.85 billion monthly active users as of the
| first quarter of 2021
|
| > 1.8 billion of Facebook users (66%) use the app on a
| daily basis
|
| Say 1 database could store the data + handle the
| reads/writes for 100,000 users (literally just guessing)
|
| 2.85 billion monthly active users / 100k users per server =
| 28,500 servers/"instances" for MySQL
|
| That's not even counting Kubernetes pods (if they use that)
| running Docker containers (if they use that) for their API
| / content serving.
|
| I wonder what their average transactions per second looks
| like.
|
| For Twitter:
|
| > Every second, on average, around 6,000 tweets are tweeted
| on Twitter, which corresponds to over 350,000 tweets sent
| per minute, 500 million tweets per day and around 200
| billion tweets per year.
|
| > As of the first quarter of 2019, Twitter averaged 330
| million monthly active users
|
| Twitter has 11.5% the MAU of Facebook (330m of 2.85b)? I
| don't know if 1 tweet can be extrapolated to 1 Facebook
| like/comment/share/post but if it did, 6k tweets per sec *
| 8.7x the user base = 52.2k writes per second?
|
| Found this:
|
| > Facebook now has about 30,000 servers supporting its
| operations, hosts 80 billion photos, and serves up more
| than 600,000 photos to its users every second.
| packetslave wrote:
| Kubernetes, no. Twine (aka "Don't call it Tupperware"),
| yes
|
| https://engineering.fb.com/2019/06/06/data-center-
| engineerin...
|
| Also, that "30,000 servers" number is off by... a lot. No
| comment on by how many orders of magnitude. :-)
| dweekly wrote:
| Yep. Here's how I'd napkin math it.
|
| Facebook has purchased 6GW of renewable energy to power
| its datacenters [1]. Its datacenters have a PUE of about
| 1.06 last they published (they unfortunately seem to have
| taken their dashboards down) so that's about 5.6GW of
| electricity actually powering real load. Some of that
| will power the networking fabric so we could
| conservatively estimate 5GW of compute load. The OCP
| Yosemite v3 platform power budget is a max of 1.5kW [2].
| Also conservatively, let's assume a real average 1kW
| load. I'll leave the division as an exercise for the
| reader to estimate the number of servers they operate.
|
| 1 https://www.google.com/amp/s/www.euronews.com/green/amp
| /2021...
|
| 2 https://www.opencompute.org/documents/ocp-
| yosemite-v3-platfo...
| MuffinFlavored wrote:
| 5 Gigawatts to Kilowatts = 5000000
|
| 5000000 kilowatts / 1kW per machine = 5,000,000 machines
| internationally?
| fowl2 wrote:
| GitHub also seems to use MySql, as recently as 2020[1]
|
| [1] https://github.blog/2020-05-20-three-bugs-in-the-go-mysql-
| dr...
| derekperkins wrote:
| Still do
| daniel_iversen wrote:
| We used MySQL at Dropbox too, and they probably still do.
| pbreit wrote:
| Any idea what Facebooks stores in MySQL? Core user database?
| kevinventullo wrote:
| To a first approximation, everything.
|
| Most product engineers interact with it by querying TAO,
| which you can read about in various blog posts, but is
| basically a cache in front of MySQL (this is a vast
| oversimplification).
| notacoward wrote:
| Hi, Kevin. :) As of the time I left, the bulk storage
| system I worked on - images, video, machine-learning
| datasets, backups of other things, etc. - held most of the
| bytes at Facebook. The MySQL/TAO complex had orders of
| magnitude more addressable objects, undoubtedly handled
| many times more requests and bytes _in flight_ per second,
| but when it comes to bytes _at rest_ they 're not even
| number one.
| notacoward wrote:
| Basically _all the small stuff_. Every user 's identity,
| preferences, history, and connections. If you post text,
| that's there too. If you post media, that's somewhere else.
| Connections are the hard part, because it's a gigantic graph
| with the potential to make every query - e.g. to construct a
| user's timeline - hit vast numbers of servers. There's all
| sorts of special sauce to do with partitioning, caching, etc.
| to absorb as much as possible of that load before it hits the
| MySQL servers, but it's still a gigantic service even by
| Facebook standards.
| tpetry wrote:
| So there has not been any major improvement in MySQL for 3
| years? There hasn't been released any minor version? Can
| someone explain the current state more?
| PeterZaitsev wrote:
| No. MySQL 8 changed its engineering approach with features
| released in minor releases.
|
| In this regard it is like MacOS X or Windows 10, whatever you
| prefer where features are shipped without major version
| change
| tpetry wrote:
| Sure, and this is great. But The newest release is 8.0.26,
| so there hasn't been any minor release yet - in 3 years.
| And i can't find anything about new MySQL features or
| planned features except the big 8.0 release.
| buchanmilne wrote:
| MySQL GA dates: 5.5 3 December 2010 5.6 5 February 2013
| 5.7 21 October 2015 8.0 19 April 2018
|
| The average seems to be about 2.5 years between minor
| releases, so this doesn't seem unusual.
|
| Usually new minor versions are only released if there
| are: * Changes to database behaviour defaults (e.g.
| default sql mode) * Deprecation or removal of features *
| Compatibility changes (changes to replication formats)
|
| New features are fine, as long as they don't introduce
| regressions in other areas.
|
| Even MySQL 5.7 received some good improvements in the
| past 18-24 months (not just bug fixes).
|
| I feel this is a really pedantic complaint from someone
| who either doesn't use MySQL, or wants to hate it for a
| silly reason.
| PeterZaitsev wrote:
| There are new features introduced in 8.0.26 as in .25 .24
| etc before that check out release notes.
| fipar wrote:
| There have been several minor releases in the past 3
| years: https://dev.mysql.com/doc/relnotes/mysql/8.0/en/
| taywrobel wrote:
| In semantic versioning parlance those would all be
| considered "patch" releases, not a "minor" release. The
| minor version segment in a semantic version is the 2nd
| number, so most people would consider the next "minor
| release" for MySQL to be 8.1.0.
|
| If following the guidance set out at https://semver.org,
| a patch release wouldn't add any new functionality, it
| would just address bugs in a backwards compatible way. A
| minor version would introduce new functionality in a way
| that doesn't violate backwards compatibility.
|
| Quickly scanning the release notes for some of the MySQL
| 8 versions, they seem to be generally sticking to this;
| most of the versions fixed some bugs or unintuitive
| behavior, but it doesn't look like much new functionality
| has been introduced since 8.0.4, the last release
| candidate for the major version bump.
| umanwizard wrote:
| SemVer is a relatively recent proposal that the vast
| majority of software does not follow. Calling the three
| parts of a three-part version "major", "minor" and
| "patch" is a SemVer thing, not a universal standard.
| 101011 wrote:
| Worth calling out here that SemVer has been around for a
| decade now, so "recent" is relative
| rtsil wrote:
| For some of us, that is recent.
| JimDabell wrote:
| SemVer did not introduce major / minor / patch, that was
| standard usage in free and open source software for
| decades beforehand.
|
| Example from 20 years ago:
|
| > The most popular version numbering scheme stems from
| the <major>.<minor>.<patchlevel> scheme
|
| --
| https://ask.slashdot.org/story/01/01/05/0054230/version-
| numb...
|
| Example from 1997:
|
| > Python versions are numbered A.B.C or A.B. A is the
| major version number -- it is only incremented for major
| changes in functionality or source structure. B is the
| minor version number, incremented for less earth-
| shattering changes to a release. C is the patchlevel --
| it is incremented for each new patch release.
|
| -- https://web.archive.org/web/19970501012343/http://www.
| python...
| wongarsu wrote:
| There's also Windows with it's A.B.C.D or
| major.minor.patch.build version scheme. Not sure how far
| back that goes, but it clearly predates semver as well.
| umanwizard wrote:
| Fair enough, you're right that it's more widespread than
| just SemVer (though IMO still not an ironclad standard)
| PeterZaitsev wrote:
| MySQL does not follow semantic version and there have
| been features introduced in minor released including some
| incompatible changes as you can see for example https://d
| ev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-24....
| Tostino wrote:
| Yeah, no thank you. I'll stick with Postgres.
| fastball wrote:
| That's still a patch release not a minor release.
| bonzini wrote:
| If it does not follow semantic versions, it is just a
| third-level version number. It's like Linux's 2.6.x
| series.
| mumblemumble wrote:
| I'm coming to realize that semver is a lot like scrum.
| Everyone knows about it, lots of people like it, plenty
| of people want to use it, nobody seems able to move past
| arguing about how it's supposed to work and actually get
| a working implementation off the ground.
| evanelias wrote:
| Many major new features have been introduced in 8.0 point
| releases. Some examples:
|
| * 8.0.12 added INSTANT algorithm for adding new columns
| to a table
|
| * 8.0.13 added DEFAULT expressions (column default values
| may now be any arbitrary expression, not just a constant)
|
| * 8.0.16 added CHECK constraints
|
| * 8.0.17 added the CLONE plugin for rapid physical
| copying of a db
|
| * 8.0.23 added INVISIBLE columns (excluded from SELECT *)
|
| These are just a handful among many others... basically
| MySQL 8 does not follow SemVer at all, nor does it claim
| to. (Although personally as a tool developer in the mysql
| ecosystem, I selfishly wish they did!)
| dijit wrote:
| That's pretty scary, isn't it?
| evanelias wrote:
| No, since MySQL 8 doesn't ever claim to use SemVer. They
| have an explicitly stated policy of including features in
| point releases. Why would it be "pretty scary" that a
| non-SemVer software doesn't use SemVer?
|
| SemVer is an arbitrary versioning scheme, not a universal
| standard. It's the operator's responsibility to
| understand the versioning scheme prior to upgrading.
| thayne wrote:
| > Why would it be "pretty scary" that a non-SemVer
| software doesn't use SemVer?
|
| Is there another level of patches below point releases
| for backported security patches, or are your options to
| upgrade to something that adds new features and may be
| backwards incompatible, leave your system vulnerable to
| known security wholes, or patch it yourself (or maybe
| depend on your distro to do itl
| buchanmilne wrote:
| > Is there another level of patches below point releases
| for backported security patches
|
| No, because its unecessary (in my experience)
|
| > or are your options to upgrade to something that adds
| new features and may be backwards incompatible
|
| It won't be backwards incompatible, otherwise they would
| leave the change for a new minor release.
|
| Why would you not want a new feature (e.g. the new "clone
| plugin") if it gives you value now without any migration
| required (because of incompatibilities that may arise if
| all new features are left for minor versions which are
| reserved for incompatibilities)?
| thayne wrote:
| > No, because its unecessary (in my experience)
|
| Assuming that is true, this sounds like a normalcy bias
| to me (https://en.wikipedia.org/wiki/Normalcy_bias).
|
| > Why would you not want a new feature (e.g. the new
| "clone plugin")
|
| Because features can introduce bugs or performance
| regressions. Sure you can say that's rare, but I still
| might not want to take that risk with some of the most
| critical infrastructure in my app.
| evanelias wrote:
| In practice, this is rarely a problem unless you're a
| cloud vendor with a DBaaS product; the most common
| concern there tends to be bugs that cause server crashes.
| But even that is fairly rare, and those platforms (AWS,
| GCP, Azure, Alibaba, etc) generally maintain custom patch
| sets anyway.
|
| As far as I can recall, the vast majority of modern MySQL
| CVEs have been low severity and/or require the attacker
| to already be in your private network.
|
| For sake of comparison -- does Postgres backport every
| security patch to every major.minor tree? And aren't HA
| Postgres upgrades generally more difficult than MySQL
| ones since logical replication is less widespread in the
| PG landscape?
| anarazel wrote:
| > For sake of comparison -- does Postgres backport every
| security patch to every major.minor tree?
|
| Unless the security fix in question is not applicable to
| some branch, yes.
|
| https://www.postgresql.org/support/security/
| https://www.postgresql.org/support/versioning/
| throwdbaaway wrote:
| It is pretty scary because most of these new features or
| enhancements are buggy. By not adopting SemVer, users of
| MySQL who prefer stability (oxymoron?) have no obvious
| upgrade path. They could be happily using version
| 8.0.123, and then upgrade to version 8.0.126 to get some
| security fix, and suddenly encounter a bunch of
| functionality and/or performance regressions.
|
| Some examples...
|
| WL#10310 Redo log optimization:
|
| Feb 2018, initial commit 6be2fa0bdbba, landed in 8.0.11
| (Apr 2018)
|
| Jun 2018, crash regression fix commit 270d18368650,
| landed in 8.0.13 (Oct 2018)
|
| Apr 2019, performance regression fix commit 75c4f7161a56,
| landed in 8.0.18 (Oct 2019)
|
| WL#5655 - InnoDB: Separate doublewrite file to ensure
| atomic writes:
|
| Dec 2019, initial commit ce14ef911, landed in 8.0.20 (Apr
| 2020)
|
| Feb 2020, data loss regression fix commit c1bc61dc7,
| landed in 8.0.20 (Apr 2020)
|
| May 2020, stall regression fix commit 00b284707, landed
| in 8.0.21 (Jul 2020)
|
| In comparison, Cassandra during the DataStax days was
| extremely buggy, but at least it tried to follow SemVer.
| So the operators can follow some simple guideline, e.g.
| >=2.0.14 is fine, >=2.1.17 is fine, >=2.2.9 is fine. Of
| course regression can still happen, but that would be an
| exceptional case.
| kaba0 wrote:
| This is a misconception that new features introduce more
| bugs than eg. patches/minor version upgrades. I can only
| bring up the OpenJDK project as example but this sort of
| change happened there as well - what used to be JDK 8.xxx
| and everyone jumped to that without second thought is the
| exact same thing as JDK 9->10. It's just management's
| backwards thinking that somehow the former is safe to
| apply while the latter is not.
| throwdbaaway wrote:
| There is certainly no hard and fast rule. In the JDK
| case, in terms of language design, my observation is that
| with Brian Goetz in charge, it doesn't matter too much
| whether it was the slow release cycle back in the pre-9
| days, or the shorter one from JDK 9 onward -- whenever a
| new feature is introduced, all kinds of compatibilities
| are being maintained at all costs.
|
| (Except Project Jigsaw of course, but I'd like to think
| that he got overruled by his boss)
|
| As for bugs, I will just quote Gil Tene from
| https://www.theserverside.com/opinion/Dont-ever-put-a-
| non-Ja...
|
| > Play with them on your laptop, but don't use a single
| feature, and wait for the LTS.
|
| Personally I think the LTS model is great. Features get
| rolled out as soon as they are ready, and then get
| stabilized after being used by early adopters. Meanwhile,
| production can stay on LTS and continue to get just the
| necessary fixes.
| evanelias wrote:
| In my experience, it is quite an exaggeration to claim
| that "most" post-GA 8.0 features or enhancements are
| buggy, especially in the general case. Yes, _some_ of
| them may have edge-case bugs or performance issues, but
| typically only affecting _some_ users with very specific
| environmental or workload situations.
|
| Operators must test before upgrading. Luckily some third-
| party software makes this easier, e.g. Percona's pt-
| upgrade or ProxySQL's mirroring feature.
|
| I'll state again unambiguously, I'm not personally fond
| of 8.0's release policy of rolling out new features post-
| GA. But I view it as an annoyance, and at least one with
| some upsides (not having to wait 2 years for a massive
| feature dump all at once) rather than being uniformly
| negative or "pretty scary". Personally I was careful
| about testing point release upgrades before 8.0, and I'm
| still careful about it now.
| dijit wrote:
| I'm not saying semver is a universal standard but this is
| _wildly_ different to everything else.
|
| Nobody expects defaults to change between patch/minor
| versions.
|
| This is another example of how MySQL does not give a
| flying F about the footguns it leaves lying around.
|
| Putting the onus on the user is not reasonable;
| especially in a world where we're trying to reduce toil
| or get rid of ops completely.
| buchanmilne wrote:
| > Nobody expects defaults to change between patch/minor
| versions.
|
| Can you provide an example of an application-side (IOW,
| not a DB-infrastructure-type variable) that has changed
| it's default in a patch version _after_ GA in 5.7 or 8.0?
|
| IIRC, all of the defaults changed in very early non-GA
| versions.
| evanelias wrote:
| In another subthread, you're (justifiably) unhappy with
| how they didn't change the strict-mode default soon
| enough; but here you're upset that some defaults changed
| too fast?
|
| Also my comment above in this subthread wasn't even about
| defaults. I only mentioned some features added, and you
| responded saying it's scary and then talking about
| defaults?
|
| You don't like MySQL, I get it. Regardless of reasons, I
| don't think your view will change, so why enter every HN
| thread about MySQL just to repetitively bash MySQL?
| What's the purpose of this?
| dijit wrote:
| The purpose, mainly is to warn people of the pain I've
| suffered. My life should be a lesson here because
| honestly it is _painful_ to realise you have lost
| monetary data and have to go hat in hand to the CTO and
| CFO to explain why.
|
| The strict mode setting should have been the default
| since the beginning. It is absolutely unthinkable that it
| wasn't, the only justification I can think of is that:
|
| a) it wasn't built with that in mind and thus was
| experimental.
|
| b) was not enabled to ease adoption (this is kinda evil,
| in my personal opinion because it teaches bad habits)
|
| c) was considered a power-user feature, which is
| ridiculous.
|
| What you've said just exemplifies the trend I've seen
| before: MySQL does not care about creating footguns, and
| people who have bought into the ecosystem like to exhalt
| that "you're holding it wrong"- which is absolutely not a
| good warning.
|
| If you listen to my rambling about the pain I've seen,
| and you still want to use it: at least it's an informed
| decision. But pretending that all is well is not ideal,
| it doesn't lead anyone to make better software or
| practices (MySQL) or better developers (because they
| don't see footguns before they've sprung some self-
| inflicted wounds on themselves)
| evanelias wrote:
| > The strict mode setting should have been the default
| since the beginning.
|
| I absolutely agree with that. But what's the point in
| talking about a terrible default that changed _nearly 6
| years ago_ , literally _over and over again_ , year after
| year?
|
| > the only justification I can think of
|
| The real answer is almost certainly "backwards
| compatibility". Same reason many things in Windows are
| non-ideal, for example.
|
| Someone made a really bad technical decision early on,
| but fixing it overnight will break things for an utterly
| massive number of paying customers, so it can't be
| rectified without a slow transition plan. _It happens._
|
| Yes, it absolutely should have been fixed on a faster
| schedule regardless, given its importance. But just
| because it wasn't, doesn't mean that the entirety of the
| piece of software is hopelessly flawed and everyone
| involved in its development is an utter cretin.
| Especially when a piece of software is multiple decades
| in age and has gone through multiple corporate
| acquisitions.
|
| > What you've said just exemplifies the trend I've seen
| before: MySQL does not care about creating footguns
|
| What I've said? What footguns? I'm asking this honestly:
| what are you referring to? Let's recap this subthread:
|
| * @taywrobel said "it doesn't look like much new
| functionality has been introduced since 8.0.4"
|
| * I listed a number of new features introduced in 8.0
| point releases, and explained that MySQL 8 is
| intentionally not doing SemVer
|
| * You said "that's pretty scary" without ever elaborating
| on which specific thing I said you find scary
|
| * I disagreed, given MySQL's stated written policy of not
| following SemVer
|
| * You mentioned something about defaults changing (???
| this wasn't even a topic I mentioned by that point in the
| subthread)
|
| Now you're talking about footguns again, but the only one
| you're mentioning with any specificity was fixed many
| years ago.
|
| > people who have bought into the ecosystem like to
| exhalt that "you're holding it wrong"
|
| Where has anyone said anything remotely like that in this
| subthread? Are you referring to my statement that the
| user should understand what's in an upgrade before
| upgrading? If so, I absolutely stand by that statement;
| it's utterly crazy to upgrade any database without even
| glancing at the release notes, let alone having a basic
| understanding of how the database vendor handles releases
| and versioning. This is operational basic practices 101,
| not about "holding it wrong".
|
| And for what it's worth, "people who have bought into the
| ecosystem" includes a massive chunk of the S&P500 using
| MySQL as a primary data store. If it's as deficient as
| you claim, why aren't all these companies going out of
| business?
| bonzini wrote:
| > _wildly_ different to everything else
|
| Linux?
| dijit wrote:
| Linux never changes backwards comparability. Linus is
| literally famous for saying as much.
| unethical_ban wrote:
| Sure, but you're assuming they follow that standard
| simply because they have x.y.z.
| dafty4 wrote:
| Thanks, HermanL and PradeepN, for sharing this migration story
| with HN!
|
| You wrote:
|
| "Despite all the hurdles in our migration path, we have already
| seen the benefits of running 8.0."
|
| May I ask what are the main performance benefits you have
| measured/noticed?
| theandrewbailey wrote:
| Anyone else notice that bad scaling and jumping on the animation?
| If they're going for an old informational film aesthetic,
| awesome!
| pradeep1288 wrote:
| this should be fixed now
| desireco42 wrote:
| So first sentence is:
|
| "MySQL, an open source database developed by Oracle, powers some
| of Facebook's most important workloads. "
|
| Oracle didn't develop MySQL, they bought it... Am I nit-picky
| here, opinions?
|
| Otherwise, congrats on shipping!
| ksec wrote:
| I would have had problems too if this was early 2010s, but I
| mean Oracle has been developing MySQL for more than a decade
| now, so I think it is fair to say "developed by Oracle".
| suyash wrote:
| Go MySQL - solid open source database rocking for the last almost
| 3 decades!
| hodgesrm wrote:
| Well...I love MySQL but there was a time when it was anything
| but rock solid. It was pretty painful until InnoDB replaced
| MyISAM in 2010 as the default storage engine. After that it
| became rare to lose data.
| dijit wrote:
| Except in cases where the constraints may be violated and it
| just inserts the next nearest thing.
|
| Or the times where the transaction isolation just abruptly
| ends and commits your transaction midway; leaving no chance
| of a rollback.
|
| Or when you alter a column to a new type and the cast doesn't
| work in a few cases and just leaves corrupted garbage on
| every row.
|
| Or when the replication misses a few transactions and your
| replica now looks different to your primary.
|
| (These aren't hypothetical, these are things I've seen)
| derekperkins wrote:
| Have you seen any of those since 5.7?
| neop1x wrote:
| I love MariaDB/MySQL. I had only like 2 crashes and 1 broken
| migration over the decade of it in production. It very powerful
| if you design tables and indices right. Postgres may be great
| too but I started with MySQL and I don't want to work with more
| DBs in parallel as there are quite a lot of differences for
| some queries. And I don't have capacity to write all queries in
| two forms. :D ORM is universal but doesn't always do queries
| optimally. And fulltext works completely differently between
| the two, for example.
| chovybizzass wrote:
| I wonder if FB could run on sqlite
| dpcx wrote:
| What I take away from this is that they are (or were, until
| recently) still running production systems on MySQL 5.6...
| gregoriol wrote:
| Some migrations are very hard!
| justapassenger wrote:
| There are production systems that run on 30+ years old
| software. Upgrade for a sake of upgrade is often a wrong
| decision.
| filmgirlcw wrote:
| Yeah, but a highly-customized version of 5.6 with its own
| storage engine. It's not like it was off the shelf MySQL 5.6.
| rajandatta wrote:
| Really well done article. In-xeptjh treatment of a problem that
| few will actually tackle - a migration at Facebook's MySQL scale.
|
| Would be really good to see how long the project took and the
| migration speed they achieved.
| thebouv wrote:
| Only commenting to say their site makes me mad cause it breaks
| the browser history. I click the link, go to the fb engineering
| site, and the back button has no history to go back to HN. Dark
| Pattern UX -- gross.
| biesnecker wrote:
| Works fine for me in Chrome 91.0.4472.164 on a Mac.
| joepmath wrote:
| Same here. Only in firefox (using v90.0)
| Semaphor wrote:
| Can't confirm. It works exactly as expected in Firefox (private
| mode because FB links open in the FB container otherwise), so
| maybe it's some extension you have installed? FB Engineering
| links are usually pretty clean, and they don't even have a
| reason to try and keep you there.
| mrcarruthers wrote:
| Yeah if you're using Firefox and you have the FB Container,
| it's working as advertised
| forgotpwd16 wrote:
| Tried with clean profile Firefox and Chromium. Back button
| works fine with both.
| takeda wrote:
| Perhaps it's due to me being in the PostgreSQL camp, but what
| happened to MariaDB? I was convinced that's where MySQL
| developers went and that's where development proceeded, while
| MySQL was just half dead, because Oracle did not care about it.
| unknownOrigin wrote:
| What happened to MariaDB? Not much, but unlike Netbeans (which
| is a complete disaster under Apache... a dying project
| maintained by a skeleton crew), it still has a commercial
| entity behind it, so it's trucking along relatively nicely. I
| mean, it's not like Oracle is investing tremendous amounts of
| time and money into MySQL either... there are still many, many
| things in the MySQL ecosystem that are second class citizens
| when compared to the big boys. Features like materialized views
| are yet to materialize etc.
| hans_castorp wrote:
| > but unlike Netbeans (which is a complete disaster under
| Apache
|
| I disagree with this
| pjmlp wrote:
| Well, when everyone and their dog is buying InteliJ licenses,
| most people that still care about doing FOSS work for Java
| IDE end up contributing to Eclipse instead.
|
| Which is a pity, Netbeans is my favourite IDE and still has
| features hardly replicated in other Java IDEs, including the
| InteliJ Ultimate edition.
|
| JetBrains would rather sell two licenses than allow for
| integrated development and debugging of native methods.
| Rapzid wrote:
| One thing of extreme value to come out of MariaDB is the JDBC
| driver that explicitly allows you to bundle it wherever you
| want.
| lucideer wrote:
| That was the expectation but the Oracle camp surprised everyone
| by actually giving it some attention, and meanwhile MariaDB's
| promise of being a "100% compatible drop-in replacement" came
| with more and more caveats with each release (as is to be
| entirely expected with any fork).
|
| When you consider the network effect of all the stacks already
| heavily invested in MySQL, all Oracle really needed to do was
| put in a modicum of effort to MySQL to stave off the MariaDB
| migrations (and from my understanding they've put in quite a
| bit more than a modicum).
|
| Lastly, Postgres is an interesting player. Migrating from MySQL
| to Postgres might be a scarier prospect than to MariaDB, but
| since the Oracle acquisition a lot of MySQL-veterans were
| looking for something non-MySQL for _new_ projects and in that
| I get the sense Postgres beat out MariaDB.
| munk-a wrote:
| I carried out a (quite ill-advised in retrospect) migration
| from MySQL to Postgres on a mature codebase with absolutely
| no automated test coverage and it went pretty smoothly. We
| had some bumpy performance directly after the production
| migration but we were able to reach parity after about two
| months of observing and adding targeted indices.
|
| I absolutely adore Postgres's tweaks to the SQL language,
| that dialect is amazing and has served us extremely well.
| And, since we passed pg10 a while back, the performance
| tuning you can do on it is pretty amazing.
| lucideer wrote:
| I am very much an example of one of those previously MySQL
| devs that now uses Postgres for new projects, though I've
| never done a migration between the two.
| code_duck wrote:
| I had an app with a fairly complex and large set of tables
| and data and migrated it from mySQL to pg with very little
| trouble. It was a few years back, but from what I recall a
| re-wrote a few queries in my data layer and that's about
| it. Much easier than it was to setup nginx to take over
| some Apache duties.
| tapoxi wrote:
| We also did a MySQL 5.7 to PG 9.6 migration (thanks to
| https://pgloader.io/) and we got some pretty immediate
| speed improvements (2-5x).
|
| We shot ourselves in the foot a few weeks later with some
| poorly written RLS policies, but it was overall a smooth
| experience.
| dijit wrote:
| pgloader is great.
|
| Sadly for me when I tried to make a PoC of Postgres for
| our eCommerce platform I found that we had violated our
| constraints and lost data.
|
| That was the day I went from MySQL agnostic to MySQL
| hating.
|
| It largely doesn't matter that the defaults could be
| configured to be more strict. We live in a world where
| people are trying to avoid hiring sysadmin and that
| database was set up by a developer. It should not have
| been a default.
| evanelias wrote:
| > It largely doesn't matter that the defaults could be
| configured to be more strict. We live in a world where
| people are trying to avoid hiring sysadmin and that
| database was set up by a developer. It should not have
| been a default.
|
| _Everyone_ agrees with this. That 's why they changed
| the default nearly 6 years ago.
|
| Yes, the old default was terrible. But there's no way to
| change the past, so why complain about it for years and
| years?
|
| All versions with this bad default have hit end-of-life,
| what more can be done?
| twic wrote:
| > I absolutely adore Postgres's tweaks to the SQL language
|
| Do you mean additions to the SQL standard, or bits of the
| standard that MySQL doesn't implement, or something else?
|
| The reason i ask is that i also enjoy writing SQL for
| PostgreSQL, but as far as i know, i am sticking to standard
| SQL. Perhaps there are things i'm missing, or things i like
| which i haven't realised are nonstandard!
| munk-a wrote:
| Personally, back before JSON became a standard thing - I
| made heavy use of HSTORE to clean up some insanely
| cludgey string packing BS we'd been doing - leveraging
| recursive CTEs using the UNION definition allowed us to
| move a fixed tier system into a self-referential one that
| saved us a lot of headaches as well.
|
| Other folks have mentioned :: instead of CAST AS (which
| is really nice when you need to write it a bunch of
| times) but it's mostly the diverse built-in function set
| and constraint support that keeps me an enthusiastic
| Postgres supporter.
| paulddraper wrote:
| > as far as i know, i am sticking to standard SQL
|
| PostgreSQL documentation is very good about comparing
| each feature to the standard.
|
| For example, the SQL standard specifies that triggers
| fire in the historical order they are added to the table.
|
| PostgreSQL chose to fire them alphabetically.
|
| Technically nonconformant, but in practical terms, far
| better.
| dijit wrote:
| Also the c++ bindings are very mature.
|
| This was surprising for my old tech lead who was sheepish
| on moving away from MySQL.
|
| He would later refer to the MySQL bindings as "brain
| dead"
| ipaddr wrote:
| Why would alphabetically be better? Wouldn't you want
| first in first out?
| throwaway2037 wrote:
| Why was this down-voted? It is a good question. I thought
| exactly the same myself. And the reply from @setr is
| helpful. (Hat tip) I never thought about the impact of
| dump/restore!
| Pamar wrote:
| Not only what @setr answered but consider also that by
| having a consistent ruling you can actually _design_ or
| _alter_ the order of triggers just by naming them
| according to a specific scheme.
|
| It just makes easier to understand what happens when
| triggers are invoked, without having to actually force
| specific cases by direct experimentation.
|
| I did not even know that the standard prescribed a FIFO
| order, looks like a pretty bad idea to me, actually.
| paulddraper wrote:
| The problem is that it doesn't leave you a great way to
| adjust order. It's just kinda weird.
|
| (To be fair, table columns share this characteristic.)
| marcosdumay wrote:
| > To be fair, table columns share this characteristic.
|
| The table columns order is meaningless, but the triggers
| order is very relevant.
| setr wrote:
| Alphabetical makes it so two databases with the same
| triggers have the same behavior. FIFO breaks that,
| because now it depends on trigger construction order (eg
| when dump/restoring) and is totally invisible to the
| user.
|
| Additionally, modifying a trigger by drop/creating it
| would change behavior, unless you also drop/create
| _every_ trigger (in the original order)
| ipaddr wrote:
| Thank you most important thing I learned today.
| jolux wrote:
| It means that the order they fire is imperative and
| determined by how you run your migrations.
| eatonphil wrote:
| MySQL does not conform to the standard in a number of
| ways. On the one hand, thankfully you can turn on ANSI
| MODE. On the other hand if you're using MySQL you've
| probably got a bunch of devs used to their non-ANSI SQL
| and turning on ANSI MODE may break stuff they do.
|
| https://dev.mysql.com/doc/refman/5.7/en/sql-
| mode.html#sql-mo...
| jacobmischka wrote:
| I assume they mean non-standard additions in PostgreSQL.
| A lot of the very handy pg functions are non-standard, as
| well as the rich text matching capabilities, many data
| types, and even the RETURNING clause, I believe.
| felixfbecker wrote:
| There is definitely stuff that is non-standard, like the
| casting with :: in addition to CAST AS.
|
| But it's true that PSQL sticks very closely to the
| standard or only adds on top of it, while MySQL is very
| uncomformant.
|
| Funnily enough, I actually first used Postgres over MySQL
| a long time ago when I moved from PHP to NodeJS, and was
| annoyed that MySQL uses backticks for identifiers while
| PSQL uses SQL standard double quotes, because in JS
| multi-line strings (for SQL queries) are also use
| backticks and I was annoyed by escaping. Sometimes it's
| the little things...
| dmitryminkovsky wrote:
| You don't think the backticks lend a nice old school
| vibe?
| petergeoghegan wrote:
| It is practically essential for any RDBMS to provide non-
| standard statements and features. For example, even
| CREATE INDEX isn't in the SQL standard, since the SQL
| standard deliberately avoids talking about what it
| considers to be implementation details. This is probably
| a good goal in many ways. But what counts as an
| implementation detail is open to interpretation.
|
| You could say the same thing about the isolation levels.
| They are supposed to be platonic ideals, free from any
| implementation baggage. But the reality is that you can
| tell that the people that originally defined how they
| work were mostly (perhaps entirely) thinking about old
| school two-phase locking.
|
| I'm not a critic of the standard -- it's imperfections
| (which are arguably contradictions) reflect real world
| differences that are hard or impossible to resolve. Just
| pointing out that this is how it is.
| munk-a wrote:
| Oh, it's totally fair to criticize the standard. Those
| clowns lock their deliberation and standard definitions
| behind huge paywalls.
|
| The best way to learn the SQL standard is to read the
| Postgres docs since they clearly call out non-compliances
| - there is no other easy to access and clear explanation
| of the SQL standard.
| dzolob wrote:
| Any "platonic ideals" mention outside its natural scope
| deserves attention.
| gunapologist99 wrote:
| If you use any of the really amazing JSON or Array stuff,
| you're stepping outside of the standard.. but it's worth
| it!
| mdaniel wrote:
| The word "any" there is mildly misleading about the JSON
| things, since there is now SQL/JSON which PG supports:
| https://www.postgresql.org/docs/13/functions-
| json.html#FUNCT...
|
| It would be horribly painful to stick to _just_ SQL/JSON
| for manipulations, and I have absolutely zero idea what
| the coverage is in other platforms, but portability isn't
| completely off the table like it used to be
| dkersten wrote:
| I absolutely love Postgres' JSON and array support. Its
| saved me from needing a document database in a few cases,
| which drastically simplified the overall deployment (one
| DB instead of two, or trying to represent relational
| stuff in a document DB). I really like JSON for stuff
| that is user defined, that I don't need to query into,
| but postgres' support means I still can should I ever
| need to. Which has happened once or twice for debugging.
| fn1 wrote:
| > but postgres' support means I still can should I ever
| need to.
|
| And since postgres supports indices on arbitrary
| expressions you can use CREATE INDEX for the json-queries
| in your WHERE:
|
| CREATE INDEX ON publishers((info->>'name'));
| dkersten wrote:
| Yes, Postgres' jsonb and array support lets you not only
| query into these fields, but also, like you say, index
| into them too. Its really the best of both worlds: you
| get a great solid relational database, but if you need
| document database features, you don't really lose out on
| anything by using json, except that the queries are a bit
| more ugly. I'm super happy with postgres these days, its
| rock solid and full of fantastic features.
| 1propionyl wrote:
| 100%. I've dealt with many small-to-medium sized projects
| where the usual approach would have been some form of EAV
| mess.
|
| Just using a BSON column for such attributes hits the
| Pareto 80/20 sweet spot, _and_ I can query against them
| much more easily than in an EAV schema (far fewer joins).
|
| They're great for when the schema is mostly normalized,
| but there's just a little bit of denormalized stuff that
| doesn't warrant completely architecting the schema around
| it.
| scrapcode wrote:
| I can see the business of open source software support coming
| to play as well. I went from freelance development to more of
| a govt contracting role and just recently pivoted my career
| back towards tech. I never saw this side of the business but
| it is thriving from where I sit.
| tomxor wrote:
| > meanwhile MariaDB's promise of being a "100% compatible
| drop-in replacement" came with more and more caveats with
| each release
|
| Anecdata: I've been using mariadb as a drop in mysql
| replacement for over 5 years in production, and it's been
| working seamlessly.
|
| I'm no DB expert, and perhaps my use case isn't complex
| enough or written in a highly mysql dependent way - but I
| felt like _someone_ should chime in since there 's not much
| positivity toward mariadb in this thread so far. I'd be
| interested to hear from some other people with _actual_
| mariadb experience.
| lucideer wrote:
| I think it's still pretty good. I ran into some issues with
| json support some years ago, but querying json may be an
| edge case (as opposed to storing text & creating your own
| indices)
| Shorel wrote:
| I have a lot of stored procedures and functions that were
| converted from Oracle to MySQL, and they also work just
| fine in MariaDB.
|
| I guess any actual incompatibility is in fringe cutting
| edge features, which is where both databases have diverged.
| cknoxrun wrote:
| We migrated a high-traffic, complex site with a huge amount
| of data to MariaDB about 5 years ago as well, and it's been
| solid so far. We use Galera as well, which has performed
| and scaled really admirably.
| sltkr wrote:
| I was curious so I looked up the differences here:
|
| https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility/
|
| It doesn't sound too bad. The incompabilities seem to be
| mostly stuff like not being able to generally use replication
| from MySQL to MariaDB or vice versa. But I would guess that's
| not a very common case anyway: most users are either
| developers that want their software to work with both MariaDB
| and MySQL (which can be achieved by sticking to the very
| large intersection of supported features), or they are users
| that want to switch from MySQL to MariaDB (which seems well-
| supported). Running a mix of MariaDB and MySQL servers and
| expecting to be able to replicate between them seems like a
| particularly uncommon setup (though maybe it's useful during
| a migration).
| hodgesrm wrote:
| Binlog incompatibility makes migration a lot more difficult
| at scale.
| hellcow wrote:
| The issues I encountered while using MariaDB in place of
| MySQL were subtle -- most things worked as expected, which
| makes the issues so much more unexpected. But at the end of
| the day, the same migrations produced a different schema.
|
| Arguably MariaDB made the correct decision to fix some
| MySQL 5.7 bugs that we were unwittingly relying upon, but
| the different behavior caused issues for us all the same.
|
| Rather than worry about MySQL vs MariaDB and whether the
| one I need is packaged and available for my distro/OS of
| choice, I reach for Postgres on new projects.
| PeterZaitsev wrote:
| I think this is a key difference - where MariaDB
| continues to evolve MySQL 5.6/5.7 codebase Oracle chose
| the "big leap" with MySQL 8 - introducing data dictionary
| and many other internal architecture changes. This is one
| of the reasons migration to MySQL 8 can be more painful
| than releases before it but for good reason
| skrebbel wrote:
| Anyone on the in-crowd who can share a bit about Oracle's
| motivations here?
|
| I recall a Bryan Cantrill talk about how Oracle is this
| completely amoral machine that just wants to make money, and
| how that drives their every decision [0]. Is this no longer
| the case? Or is it, and if so, how does improving MySQL and
| giving it away for free making Oracle money? Or was it always
| hyperbole and Oracle is improving MySQL because they want to
| be nice?
|
| [0] https://www.youtube.com/watch?v=-zRN7XLCRhc&t=2047s, that
| link starts at the beginning of what's popularly known as
| "the lawnmower rant", it's delightful.
| kaba0 wrote:
| Oracle, compared to their image is quite technically adept
| -- they did a splendid job with OpenJDK and they have
| plenty of long-term R&D projects like GraalVM.
| skrebbel wrote:
| Yeah so, all of that seemed to happen _after_ that
| Cantrill talk. How did Oracle suddenly turn from a pure
| license contract squeezing company into a tech leader
| again? I mean, they had to hire great engineering talent
| to do all this. And convince them to work for _Oracle_.
| What 's the narrative I'm missing?
| kaba0 wrote:
| I'm not familiar with the internal stucture of Oracle,
| just tired of the imo unfair bashing of them (at least
| compared to all other companies).
|
| But I guess they must have had at least sane management
| that tried to keep the existing mindshare of acquired
| projects (as far as I know many people went from Sun to
| Oracle directly for example), as well as finding new
| talent. And also, the acquisitions made sense and were
| done at a comparatively good price so I guess they are
| good lawnmowers with long-term plans (but terrible PR)?
| ksec wrote:
| >Yeah so, all of that seemed to happen after that
| Cantrill talk.
|
| The talk was in 2011, Oracle purchased Sun in 2010. So
| basically their continuous work on OpenJDK has nothing to
| do with the talk at all.
| brianwawok wrote:
| No it's 100% still correct.
| takeda wrote:
| I guess I'm cynical, but to me looks like Oracle renewed
| interest in MySQL after PostgreSQL gained momentum. MySQL
| is not really a competitor to Oracle but PostgreSQL
| overlaps with both of them.
| beberlei wrote:
| Oracle is selling Enterprise features and Support, i am
| sure they make a lot of money with MySQL.
| zdragnar wrote:
| Oracle paid a _lot_ of money to buy Sun, and basically
| ended up with two products- selling support for Java and
| MySQL. Letting MySQL rot would have been pretty bad for
| their pocket book (especially as others have noted that
| postgres has been cutting into their market heavily).
|
| In Bryan's defense, the lawnmower rant is pretty true- they
| are a simple company. They focus heavily on making money
| and abandoning things they aren't in a good position to do
| so from.
|
| Selling per-cpu-core licenses and support contracts, great!
| Experimenting on or converting small projects into new
| markets? Not great. Fleshing out OpenSolaris into a product
| that could rival linux? Dead end.
|
| Maybe it could have worked out, but the lawnmower didn't
| care about a product that had too few users to make money
| from. OTOH, even Facebook uses MySQL... All they need to do
| is invest enough to maintain market share to get support
| contracts.
| pjmlp wrote:
| > Experimenting on or converting small projects into new
| markets?
|
| Bringing MaximeVM out of research labs into the market as
| GraalVM.
|
| First RDMS to support Perl and Java as stored procedures.
|
| Join venture with Sun for Network Computer thin clients.
|
| > Fleshing out OpenSolaris into a product that could
| rival linux?
|
| Every OpenSolaris sales is one sale less from Unbreakable
| Linux, so in a day and age where UNIX === Linux, a
| company like Oracle has chosen what makes more sense
| monetarly.
|
| I doubt that IBM is getting lots of new AIX customers.
|
| And in the end, except for a timid offer from IBM, no one
| cared to rescue Sun.
| radicalbyte wrote:
| FANG would buy them if they had somehow managed to
| stumble forward to the mid-2010s.
| pjmlp wrote:
| Zero offers from any of them, so.
| cerved wrote:
| likely support and keeping customers from migrating to
| postgres
|
| and yes, Oracle is 110% focused on the bottom line.
| Preferably this quarter
| truffdog wrote:
| Mr. Cantrill might be engaging in hyperbole. Big companies
| are big though- lots of people in charge of lots of things,
| and often they wander off in different directions. It
| doesn't make sense to model their behavior as if directed
| by a single hive mind.
| tonyedgecombe wrote:
| Companies do have a culture though, the reason Amazon
| warehouse employees have to piss in a bottle can be
| traced back to Bezos and his laser like focus on
| controlling costs.
| kaba0 wrote:
| Big enough companies are the stereotypical paper clip AIs
| everyone fear, but with money. Anything seemingly moral
| action by them is simply a cost-analysis on how the
| public opinion will effect their profits.
| freedomben wrote:
| On Fedora/RHEL you get maria when you "install" mysql. I've
| never had something fail to "just work." That said I don't do
| anything crazy, so I'm sure there are cases where they don't
| work.
| z3t4 wrote:
| When I started working with databases about 20 years ago
| talking to peers it was common to have a layer in-front of
| the database so that you could easily replace the database.
| It felt stupid until I actually had to do a database
| migration... So do you still put a layer infront of the
| database so that you can for example easily change from mySQL
| to Postgress? Given there are so many open source database I
| think there is less need to do so... Or do you go all in
| vendor lock in? Or some middle ground?
| astrange wrote:
| That would be an ORM or something similar, I think. It's
| not realistic unless you're always testing with two
| databases from the start, because their implementation
| details leak through.
|
| It can be useful if your program is simple enough to work
| with both MySQL and SQLite, but often you should just stick
| with one DB and get to use all its features.
| davecanderson wrote:
| [ServiceStack OrmLite][1] does just that, all our
| development is done locally using SQLite but then
| deployments use MySQL. We do fall into the 'program is
| simple enough' category though.
|
| [1]: https://github.com/ServiceStack/ServiceStack.OrmLite
| cubano wrote:
| I believe you are talking about the Microsoft ODBC API
| which was language and platform agnostic data acquisition
| layer. Early versions of it had pretty bad performance
| issues if I remember correctly, but eventually it became
| somewhat useful.
|
| I believe that developers back in the 90's wanted a way to
| "drop in" various low and higher cost DBMS systems to their
| client-server based apps without the need of hiring full-
| time database sysops.
| pyrophane wrote:
| I like Postgres but still turn to MySQL for production use
| most of the time for a few reasons:
|
| * Permission management on Postgres is much more painful. *
| The need for an external connection pooler makes postgres
| more annoying to set up. * Better quality docs on performance
| tuning MySQL.
| brianwawok wrote:
| Do you really need a connection pooler for most PG
| installs? I run a SaaS company with a lot going on, and
| haven't had to touch a pooler. Average about 500 open
| connections. That's a few, but on a DB server with 200GB of
| memory it's nots really breaking anything.
| hans_castorp wrote:
| > * Permission management on Postgres is much more painful.
|
| but way more flexible and powerful
|
| I only work in the Java world where all application servers
| have the connection pooling built in, so I never had the
| need for an external connection pooler.
| skeletal88 wrote:
| I joined a company where most stuff is done in Java and
| tried to explain to colleagues that sn external
| connection pool would ne good. They didn't know such
| things existed at all, and then refused because in their
| mind the application-specific intetnal pools were all
| they needed. Which was and is totally wrong - the
| application server internal pools, like HikariCP only
| share connections between threads inside the server. What
| if now we have 4 instances of the same server running
| (because of a deployment, where 2 new ones are spawned,
| started and the 2 old ones aren't taken down until the
| loadbalancer determines that the new o es are running ok.
| Now you need 2x the number of connections available than
| you would ususlly need. Once we had run out of
| connections when for some reason aws went crazy and
| started deploying too many instances of the application.
| agumonkey wrote:
| sounds like their java strategy until recently
| mathnode wrote:
| MariaDB is massive now.
|
| mysql and MariaDB diverged from versions 5.6 and 10
| respectively, although MariaDB can and does still merge some
| fixes from mysql; oracle don't do that.
|
| MariaDB and the original InnoDB developers are now doing heavy
| rewrites with big performance gains, replication options like
| multi-source, spider for sharding and galera are all built in
| and actively maintained.
|
| Oracle attempted to extort anyone trying to licence a mysql
| based product(rip infobright), so infinidb and clustrix, now
| respectively called columnstore and xpand, have new homes with
| MariaDB and are core parts of the system. Online schema
| changes, easy horizontal scale (huge numbers that make db2 and
| oracle sweat), s3 backed tables, non blocking backups,
| flashback, versioned tables.
|
| The list goes on, and many industry veterans from ibm, oracle,
| sybase etc are finding a home there and helping MariaDB grow.
|
| There are new public developer resources and the connectors are
| feature rich and well loved. There have been a few free online
| conferences during the pandemic, and MariaDB now has its own
| place at fosdem not just bundled in with mysql.
|
| If you go enterprise, there isn't much out there as accessible
| as maxscale for transaction failover!
| buchanmilne wrote:
| > replication options like multi-source
|
| MySQL 5.7 supports multi-source replication:
| https://dev.mysql.com/doc/refman/5.7/en/replication-
| channels...
|
| > spider for sharding and galera are all built in and
| actively maintained.
|
| Galera is available in MySQL-compatible binaries from Percona
| (Percona-XtraDB-Cluster):
| https://www.percona.com/software/mysql-database/percona-
| xtra...
|
| MySQL also has their own Paxos-compliant clustering solution
| (called "Innodb cluster"), first available in 5.7, but much
| better in 8.0:
|
| https://dev.mysql.com/doc/refman/5.7/en/mysql-innodb-
| cluster...
|
| I played with Spider in MariaDB 10.3, and it didn't satisfy
| the requirements we had, and it doesn't seem to have
| progressed significantly since then.
|
| Yes, it's neat that MariaDB merged it and is keeping it
| working, but there don't seem to be many use cases where it
| ends up being significantly better than e.g. multi-source
| replication (the use cases where you're going to run out of
| disk, in most cases spider's un-implemented push-down join
| etc. make it too slow to be practical either).
|
| > Online schema changes,
|
| https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-
| op...
|
| (including "instant column add")
|
| > easy horizontal scale (huge numbers that make db2 and
| oracle sweat), s3 backed tables, non blocking backups,
|
| MySQL has had "MySQL Enterprise Backup" for a long time.
| Percona wrote a similar implementation (XtraBackup), which
| works for MySQL, Percona, and MariaDB. AFAIK MariaDB copied
| XtraBackup as mariabackup (not criticising them for doing
| this, just pointing out that Percona did the original heavy
| lifting on this).
|
| > flashback, versioned tables
|
| Yep, versioned tables are a nice feature that I don't think
| are available in any other MySQL-like DB.
|
| > If you go enterprise, there isn't much out there as
| accessible as maxscale for transaction failover!
|
| Not even ProxySQL ( https://proxysql.com/ )?
|
| To be clear, I like MariaDB, but Oracle isn't doing a bad job
| on MySQL. MySQL 5.7 is pretty solid, and they have been
| improving defaults (while leaving the option to enable compat
| settings which are necessary when migrating a large
| deployment between versions), and MySQL 8.0 has some really
| nice improvements. For some of the best features from each,
| Percona (either standard Percona 8.0, which gives features
| like MariaDB's thread-pool, more comprehensive encryption
| including redo-logs etc., or XtraDB-Cluster to go with Galera
| instead of Innodb Cluster) offers quite a few
| benefits/improvements (and is contributing a lot of
| bugfixes/improvements upstream).
|
| (My team runs quite a big installation, currently running
| Percona 5.7, and migrating to MariaDB probably won't be
| practical or justifiable considering replication
| compatibility which is a requirement to do no-downtime
| migrations)
| PeterZaitsev wrote:
| I do not think there was the time when MySQL was quite dead
| under Oracle ownership. As far as I aware Engineering team in
| particularly have only been steadily growing.
|
| It is fair to say Oracle did not invest in ease of
| Oracle->MySQL migration but they have been making a steady
| progress on making MySQL better database for modern
| applications
| MangoCoffee wrote:
| MariaDB is getting more corporate. it feel like Michael
| Widenius's MySql second act.
| thayne wrote:
| MariaDB isn't even the only alternative. There's also Percona.
| From what I can tell all three have pretty decent followings
| and are actively developed, with some cross pollination.
___________________________________________________________________
(page generated 2021-07-23 23:03 UTC)