Post B5p3GnEDx6ENdqkIng by navi@social.vlhl.dev
(DIR) More posts by navi@social.vlhl.dev
(DIR) Post #B5onUanJNcitrjiRnc by ska@social.treehouse.systems
1 likes, 0 repeats
skarnet.org will be down until further notice - it may take up to 10 days (!)I wanted to reboot the server on a newly tailored kernel to avoid CopyFail shenanigans. The shutdown failed right before rebooting (because of a small oversight in s6-linux-init that is easy to fix and will not impact anyone else). Normally that wouldn't be a problem: log in to the web interface to the VPS administration, press the reboot button, done.Except, Gandi added 2FA since I last logged in, and did not notify their users. And they decided that by default, the verification code would be sent to the registered primary e-mail address.My registered primary e-mail address is hosted on skarnet.org. Because it was never a possibility that it would be needed for the bootstrap chain. Except that now it is, and I can't access it.This is the dumbest catch-22 in the history of catch-22s. And the recovery procedure, when you have to prove your identity when writing to the support outside of a logged in account, can take up to 10 days. And tomorrow is a holiday. Fuck me, I guess.I should have paid more attention, but the consequences of my not paying enough attention should definitely not be so vast, and I have choice words for several of the decisions that were made.
(DIR) Post #B5onUbOt7umTkHISMy by navi@social.vlhl.dev
0 likes, 0 repeats
@ska my first thought for stuff like this would be to quickly setup one of those mail services that allow custom domains for freemake the MX entries, get the recovery code, and remove the entriesthat is assuming dns access isn't also tied to the email address
(DIR) Post #B5onUbmzgI3yx2ZhwW by ska@social.treehouse.systems
0 likes, 0 repeats
@navi well, alyss is also the DNS server for skarnet.org, so. 😑
(DIR) Post #B5onUbygynQ9XK33B2 by navi@social.vlhl.dev
1 likes, 0 repeats
@ska can't that be temporarily changed w/ the registrar?i'm completely unaware of how hosting a dns server works
(DIR) Post #B5onUcBoC1ueC0BWca by ska@social.treehouse.systems
1 likes, 0 repeats
@navi Gandi is the registrar, I'd need to access my account to change that 🙄
(DIR) Post #B5onUcM5Zo8Uhszje4 by ska@social.treehouse.systems
0 likes, 0 repeats
@navi Really, the system is perfectly elegant and self-contained, and has one outside line to bootstrap from: my access to the web interface to control the VPS.And they decided to move that access into the closed loop, without notifying me.
(DIR) Post #B5onUcc2cUtdVMSTVg by navi@social.vlhl.dev
1 likes, 0 repeats
@ska i would honestly be a bit worried about putting all my eggs in a single basket like that tbh
(DIR) Post #B5onUcwFPN3kW1uc0O by ska@social.treehouse.systems
0 likes, 0 repeats
@navi What's the alternative? Paying for a second VPS hosted elsewhere? Having to manage another e-mail address?One of the points of skarnet.org is maximum independence. And really, it has worked pretty well so far: I could always recover from mistakes.Now my new recovery address will be my gmail one. Lovely. I could change and ask for some hosted address somewhere else, but that defeats the purpose of being self-reliant.Really, it all comes down to "don't cut my lifeline without telling me you fucking clowns".
(DIR) Post #B5onUdJzz43fhh1a1g by domi@donotsta.re
0 likes, 0 repeats
@ska @navi you could have added a second, lower-priority MX record, which points to some server that doesn’t even have to be on all the time, but worst-case scenario you can fetch your mail through itof course, not a solution for right now…
(DIR) Post #B5onuNekqiyaeF0gxU by ska@social.treehouse.systems
0 likes, 0 repeats
@domi @navi it's not like I expected my e-mail to ever be used as a necessary stepping stone to recover access to my e-mail
(DIR) Post #B5onuNxthYHxbbxynQ by domi@donotsta.re
0 likes, 0 repeats
@ska @navi well, now you know! ^^i think i had those thoughts when I changed my ovh mail address to the domain bought on OVH. my solution is having all my resources shared to another OVH account… and paying for the domain 5 years in advance XD
(DIR) Post #B5ooG3PIofCIDW7KT2 by navi@social.vlhl.dev
1 likes, 0 repeats
@ska honestly having the registrar somewhere else, and a backup mail account my registrar is separate from my vps provider, and doesn't have my self hosted email as primary, solely because i was afraid of messing up smth and being locked out of their dns panel with a broken mail setup
(DIR) Post #B5opfwJLvB8dh0F1Ci by lanodan@queer.hacktivis.me
0 likes, 0 repeats
@navi @ska Also reminds me of why I don't put all my domains at the same registrar, way too many stories of accounts being locked/closed/…But well I started with hosting almost everything at home, and then had to get a VPS for sending out emails and when I did that also put a secondary MX at home (and made DNS redundant around the same time).
(DIR) Post #B5p3Gn02not8vs6yhM by ska@social.treehouse.systems
0 likes, 0 repeats
@navi Spreading your dependencies is clearly a more resilient setup and you're right to do so.But the thing is, this incident also confirms me in my belief that it's a good thing for me to have control on as many things as possible, because as soon as I depend on one external entity, they fuck something up. So I want to be self-reliant even more.And I guess my external line will now be gmail, for the sole reason that they're too big to fail. Which I don't like, but I'm going to use that property.
(DIR) Post #B5p3GnEDx6ENdqkIng by navi@social.vlhl.dev
1 likes, 0 repeats
@ska oh yes, i want control of every bit i can getwhat i can't get i try to distribute and setup redundancies
(DIR) Post #B5p3J2ebZUvO3v9pia by ska@social.treehouse.systems
2 likes, 0 repeats
That was way faster than expected! Some person read my emergency ticket sent last night (from logged out), and I seem to have been convincing enough because they restarted the server for me, and now everything is working again. 🥳 Guess I won't have these 10 days of forced vacation. 😔 Now to take measures so that it doesn't happen again...
(DIR) Post #B5p3JdW2WJ3eO4R10K by navi@social.vlhl.dev
1 likes, 0 repeats
@ska i guess "please restart this vps that is hang anyway" is a easier ask to "give me access to this account"
(DIR) Post #B5p3gnJyEKeLSTpeYi by ska@social.treehouse.systems
0 likes, 0 repeats
@navi Absolutely. I had asked to either send a recovery code to my gmail account, or reboot the server. They rebooted the server. Totally fair. 😅
(DIR) Post #B5p3gnYVMIHABYdGDI by lanodan@queer.hacktivis.me
0 likes, 0 repeats
@ska @navi Also out of curiosity now that it's resolved, how did it hang?
(DIR) Post #B5p6oFcnhCKx88t35s by ska@social.treehouse.systems
1 likes, 0 repeats
@lanodan @navi https://git.skarnet.org/cgi-bin/cgit.cgi/s6-linux-init/commit/ (not pasting the permanent URL to avoid people getting caught by my LLM blocker)
(DIR) Post #B5p7QAKfoxlBijUGjw by lanodan@queer.hacktivis.me
0 likes, 0 repeats
@ska @navi Heh done similar mistakes quite few times ^^(Part of why sometimes I boot in read-only mode via qemu before rebooting)
(DIR) Post #B5p8JSdfSZoMQKjqGu by ska@social.treehouse.systems
1 likes, 0 repeats
@lanodan @navi I never have problems booting thanks to the static guarantees of s6-rc, whenever I have an issue it is shutting down because after some living, the state may be different from what I assumed it would be.But I'm learning, and incrementally fixing assumptions about state, this is only the latest case - and hopefully the last one.
(DIR) Post #B5p8kufzKExxCOujBY by lanodan@queer.hacktivis.me
0 likes, 0 repeats
@ska @navi Makes me wish I could trade, could always force a shutdown, while booting up when it's broken it's broken.
(DIR) Post #B5pA9KA0KN0n8n9qD2 by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@ska @navi yes, spread. No, not Googlemail.
(DIR) Post #B5pAEEtZKY5gG8oWR6 by skyfaller@jawns.club
0 likes, 1 repeats
@ska @navi Considering Gmail "too big to fail" is very... optimistic.Google can and will destroy anything in their control, up to and including Google Search. Look at the Google Graveyard! Is Google really a company you should trust with anything? https://killedbygoogle.com/I also use Gmail and I'm typing this from Android, but I live in fear of these becoming untenable overnight. I would spend a lot of time, money and effort to achieve stability elsewhere. Google may continue, but will Gmail?
(DIR) Post #B5pBI2FaNyNA9dUCYq by ska@social.treehouse.systems
1 likes, 0 repeats
@lanodan @navi You can always force a shutdown until it's on a remote VPS and the shutdown worked just well enough to kill everything including your sshd and your emergency getty and leave you with a brick and you're logged out of the VPS administrative account and the recovery codes are sent to the brick. 😅
(DIR) Post #B5pNWOFyYvxPuTHPu4 by AnachronistJohn@zia.io
0 likes, 0 repeats
@ska @navi Having a backup MX or a backup DNS server would give options. While mail often can’t be run on residential lines, DNS usually can.A backup MX will receive and store email to be forwarded, and you can just look directly in the mail queue to get the auth code, for instance.I completely agree that companies’ ideas about two factor are broken, are often added without asking and without configuration, and can be highly problematic during emergencies. At least you’re here, sharing with others, so others can consider whether they might be in the same scenario should a problem occur.
(DIR) Post #B5pNWOtKCdQtsVgqEi by ska@social.treehouse.systems
1 likes, 0 repeats
@AnachronistJohn @navi that's the point of the thread: so others can learn from my misadventure 😅Setting an external secondary DNS server is something I have thought about, obviously, but the answer was always "not worth it" because only this server is under skarnet.org so losing the DNS means losing access to a server that is down anyway. 🤷To have enough redundancy to avoid what happened to me, the external machine would also need to be MX and mail storage. I don't want to install that myself (duplicate work), I don't want to pay for that, and that's a little too much infrastructure to ask friends. I'd rather learn the lesson and not be an idiot next time 😁