Post 2794935 by ohthehugemanatee@fosstodon.org
(DIR) More posts by ohthehugemanatee@fosstodon.org
(DIR) Post #2793496 by lain@pleroma.soykaf.com
2019-01-09T07:36:56.994475Z
3 likes, 3 repeats
"The following is a real critical problem that we ran into here at TripAdvisor, but have yet figured out a clear way to mitigate.TL;DR:Streaming replicas—and by extension, base backups—can become dangerously broken when the source and target machines run slightly different versions of glibc. Particularly, differences in strcoll and strcoll_l leave "corrupt" indexes on the slave. These indexes are sorted out of order with respect to the strcoll running on the slave. Because postgres is unaware of the discrepancy is uses these "corrupt" indexes to perform merge joins; merges rely heavily on the assumption that the indexes are sorted and this causes all the results of the join past the first poison pill entry to not be returned. Additionally, if the slave becomes master, the "corrupt" indexes will in cases be unable to enforce uniqueness, but quietly allow duplicate values."What an absolute nightmare.
(DIR) Post #2793518 by kaniini@pleroma.site
2019-01-09T07:38:13.473490Z
1 likes, 0 repeats
@lain and this is why I run musl on servers
(DIR) Post #2793547 by bamfic@hub.spaz.org
2019-01-09T07:39:31.251927Z
1 likes, 0 repeats
@lain That's horrible.
(DIR) Post #2793583 by hellpie@weeaboo.space
2019-01-09T07:42:47.873018Z
1 likes, 0 repeats
@lain to be honest glibc is the reason I stay away from glibc where I can
(DIR) Post #2793626 by lanodan@queer.hacktivis.me
2019-01-09T07:38:51.086322Z
0 likes, 0 repeats
@lain *yawn*What about picking a libc that works better?
(DIR) Post #2793627 by lanodan@queer.hacktivis.me
2019-01-09T07:44:03.129412Z
0 likes, 0 repeats
@lain Also now I wonder if having a different locale with strcoll_l(3) on psql could break stuff too…
(DIR) Post #2793628 by lain@pleroma.soykaf.com
2019-01-09T07:44:16.567866Z
0 likes, 0 repeats
@lanodan apparently yes
(DIR) Post #2793766 by kaniini@pleroma.site
2019-01-09T07:51:20.920151Z
0 likes, 0 repeats
@lain @lanodan locales are boring just be like me and run everything on C.UTF-8
(DIR) Post #2794935 by ohthehugemanatee@fosstodon.org
2019-01-09T08:40:40Z
1 likes, 1 repeats
@lainThis is a great example of why PaaS database offerings make a lot of sense. It's also a great example of why containerized databases make a lot of sense.Ping me if you want to talk about architectures that can help. My job at Microsoft is helping big organizations solve problems just like this with OSS implementations.
(DIR) Post #2795311 by kaniini@pleroma.site
2019-01-09T09:00:41.582617Z
0 likes, 0 repeats
@ohthehugemanatee @lain I'm honestly confused as to what help you believe is needed?The thread is about a horror story that happened to some other group, not the group of people bringing happy easy to deploy fediverse software to the masses.
(DIR) Post #2797274 by ciaby@social.brokenbydesign.org
2019-01-09T09:42:19.647241Z
0 likes, 0 repeats
@lainWhy would you want to run different versions of glibc? 🤔
(DIR) Post #2797275 by clacke@libranet.de
2019-01-09T10:24:23Z
0 likes, 0 repeats
@ciaby @lain I don't get the impression that the quoted entity actually intended to do it. Small differences between deploys are not rare, and if collation behavior changes subtly from one minor to the next (which nobody should be surprised might happen), the cited mess is the result.
(DIR) Post #2798313 by eldergoth@sfbubble.net
2019-01-09T10:27:06Z
1 likes, 0 repeats
@lain I have one thing to say on that topic right now:Eeeeeeeeek!!!
(DIR) Post #2798570 by ciaby@social.brokenbydesign.org
2019-01-09T10:27:35.015353Z
0 likes, 0 repeats
@clackeYes, I was ironic 😁However, in the age of puppet / saltstack / ansible, I expect deployments to be basically identical. Getting a different libc version or locale feels weird to me... @lain @kaniini @danyspin97 @ohthehugemanatee
(DIR) Post #2798571 by ohthehugemanatee@fosstodon.org
2019-01-09T10:54:42Z
1 likes, 0 repeats
@ciabyTo me it's not weird, it's a choice. Somebody decided that inconsistencies like this are cheaper than engineering/maintaining a consistent environment. Whether they did this deliberately and visibly is another question - a shocking number of these trade-off decisions happen "by default." Still, it is a valid choice in some environments.@danyspin97 @kaniini @lain @clacke
(DIR) Post #2798584 by ciaby@social.brokenbydesign.org
2019-01-09T11:00:42.593276Z
0 likes, 0 repeats
@ohthehugemanateeComing from a Microsoft person (consistency? 🤔) that sounds very... Not consistent. Also, what are you doing on the fediverse? @clacke @lain @kaniini @danyspin97
(DIR) Post #2798585 by ohthehugemanatee@fosstodon.org
2019-01-09T11:09:49Z
3 likes, 0 repeats
@ciabyHahaha well I'm a #Microsoft employee, not a fanboy!I help MS's biggest customers solve their cutting-edge tech problems, using open source. We do it 100% for free, but only if we can contribute our work upstream (or to other relevant places). That's a long way to answer: I'm a career open-sourcer. Of course I'm on the fediverse! :)AMA if you like.@danyspin97 @kaniini @lain @clacke
(DIR) Post #2800892 by varx@infosec.exchange
2019-01-09T12:31:14Z
2 likes, 0 repeats
@ciaby I haven't worked with Postgres in high-availability environments, but I'm imagining a scenario where you're replacing the cluster with a new one with upgraded software. Presumably, the new one ends up with a different glibc version, and the migration occurs via replication.That would happen even in an ideal world of highly uniform deployments.@danyspin97 @ohthehugemanatee @kaniini @lain @clacke