[HN Gopher] When you deleted /lib on Linux while still connected...
___________________________________________________________________
When you deleted /lib on Linux while still connected via SSH (2022)
Author : todsacerdoti
Score : 116 points
Date : 2025-03-22 07:24 UTC (15 hours ago)
(HTM) web link (tinyhack.com)
(TXT) w3m dump (tinyhack.com)
| smw wrote:
| I had someone do this on an important production Solaris machine
| once, many years ago. Luckily they just moved /lib instead of
| deleting it -- and on Solaris (some of?) the binaries in /sbin
| were statically linked, including ln. Hard linking /lib back to
| the correct path was enough to recover.
| ryao wrote:
| Do you mean moving it? You cannot make a hard link for a
| directory. That is disallowed on Unix to prevent the creation
| of orphaned subtrees (i.e. ln a directory into itself and then
| rmdir the original link).
| yjftsjthsd-h wrote:
| GNU ln has this ( https://man.archlinux.org/man/ln.1#d ):
| -d, -F, --directory allow the superuser to attempt to
| hard link directories (this will probably fail due to system
| restrictions, even for the superuser)
|
| which implies that that's not quite an absolute limit. I
| don't see any comment either way on
| https://illumos.org/man/1/ln , but it's plausible that some
| version of Solaris had wiggle room; it's a terrible idea for
| obvious reasons, but there's really no hard technical reason
| why a system couldn't allow you to create hard links to
| directories.
| ryao wrote:
| The Solaris VFS is flexible enough for a filesystem to
| support this, but any filesystem that does violates POSIX
| according to a comment in ZFS. A check of the illumos UFS
| driver source code reveals that it violates POSIX by
| permitting root to do this. I wrote more about this with
| links to relevant illumos source code here:
|
| https://news.ycombinator.com/item?id=43447288
| wizzwizz4 wrote:
| There are two checks prohibiting it:
|
| * Non-root users aren't allowed to make directory hard links.
|
| * Many versions of the userspace program `ln` don't let you
| do it.
|
| But the `link` system call _can_ , at least on Solaris, if
| called by user 0. (Not sure about Linux: I tried it once and
| it didn't work, but I was doing weird things with FUSE and
| also trying to name the link `..`, so I don't know _why_ it
| failed.)
| ryao wrote:
| It is more nuanced than that. First, the Solaris 11.4
| documentation says that this is not allowed:
|
| https://docs.oracle.com/cd/E88353_01/html/E72487/link-8.htm
| l
|
| The illumos man page is less clear:
|
| https://illumos.org/man/8/link
|
| The illumos ZFS driver makes it very clear that this is not
| allowed under POSIX in a comment and explicitly disallows
| it in zfs_link():
|
| https://github.com/illumos/illumos-
| gate/blob/master/usr/src/...
|
| However, It appears that the illumos UFS driver supports
| this:
|
| https://github.com/illumos/illumos-
| gate/blob/master/usr/src/...
|
| Presumably, the Solaris 10 UFS driver also supports it (or
| supported it in older versions of Solaris 10). Given that
| someone at Oracle likely modified the Solaris man page to
| differ from the older OpenSolaris man page in illumos, I
| would expect recent versions of Solaris to disallow this on
| UFS, but someone would need to check.
|
| That said, I have to recant my previous comment. smw likely
| linked the directory, which is insane, but would have
| worked on older Solaris versions if we assume the modern
| illumos UFS driver is unchanged in this regard.
| jmclnx wrote:
| Interesting, at one time ages ago I thought Linux came with a
| static linked bash in /bin or maybe /sbin for this type of
| problem.
|
| Just checked Slackware and no more, I wonder if that is a
| casualty of the /bin /usr/bin merge ?
| ryao wrote:
| Gentoo used to include a static busybox binary for recovery
| purposes, but it was removed from the system set years ago. Now
| you are expected to use the initramfs for recovery and can
| install bust box yourself if you are concerned. In fairness,
| this particular issue would be made worse by rebooting since
| the unlinked anonymous library files would be garbage
| collected, which is bad unless you have a filesystem that has
| snapshots like ZFS and had used them prior to this.
|
| The article author did not try to recover libraries from the
| anonymous files, which is probably good considering that only a
| subset would have been in use and thus only that subset would
| be recoverable from the anonymous files (unless there are
| filesystem snapshots).
| pengaru wrote:
| once upon a time everything in /sbin was statically linked
|
| then only `sln` was statically linked (a variant of `ln`)
|
| today most distros statically link nothing and you're up shit's
| creek in this situation
| ryao wrote:
| The technical term is unlinked. The files in use by running
| processes are still anonymous files in the filesystem. They will
| not be garbage collected until the last program that mmap'ed them
| is gone. If you could find the inode number from /proc (possibly
| from /proc/$PID/maps), you should be able to use a filesystem
| specific tool to retrieve them, such as debugfs or zdb.
| abound wrote:
| I think the tricky piece is that you'd need to find that inode
| number _without using any dynamically linked libraries_ ,
| including ls and cat in the author's case. And debugfs is
| likely dynamically linked too (I just checked and it is on my
| machine)
| ryao wrote:
| Presumably, the read shell builtin can be used to read files
| in /proc. As per the original article, you can get a static
| version over the network using only bash builtins and
| overwrite a file with execute permission to be able to
| execute it.
| eitland wrote:
| Something similar happened a place I worked somewhere between 10
| and 20 years ago.
|
| Service technician did not see the . in the command
| rm -rf ./bin
|
| so he proceeded to run rm -rf /bin
|
| When that didn't work he did what everyone who knows a little bit
| Linux does and added sudo in front.
|
| I was on a terminal from the other side of the globe when the
| server suddenly started acting weird.
|
| We were able to use scp or rsync (one of then was in sbin or
| something) to get back the bits from an identical server, which
| saved me from three days of tedious work :-)
|
| In hindsight of course we should have written the docs in a way
| that would prevent this exact situation but in the beginning it
| was just the output of the history command after I had done it,
| dumped into a document with some explanations.
| ryao wrote:
| It is not the same thing, but recently, my pfsense router
| started acting weird where dns stopped working. I was bust so I
| rebooted it and it failed to boot. I ended up bringing a
| monitor and keyboard to see what went wrong and it turned out
| that suricata's logs had used 1.4TB on my 250GB SSD (ZFS zstd
| compression is awesome), causing it to run out of space a few
| years after I enabled the feature out of curiously. I wiped the
| logs, rebooted and things worked.
|
| My lesson from them is when a machine servers acting strangely,
| do not reboot and instead troubleshoot right away. Had I done
| that, I could have found the problem and had only minimal
| partial downtime (I would have had to restart dns afterward).
| Since I was too busy to do things the right way, my internet
| was out for a half hour. Your story reminded me of this, since
| you would have had a bigger headache if you had rebooted.
| hinkley wrote:
| This is the origin story for anyone who has ever set up
| alerts for 90% disk utilization on a machine.
|
| These are especially bad with services that generally grow
| their logs very slowly but when something like a net split
| happens or a server is down they generate as much log data in
| an hour as they typically do all week. So you get close to
| full and then an incident happens, and now you have two
| incidents because the screaming machine goes down with a full
| disk right after you lose your internal DNS server or what
| have you.
| 3eb7988a1663 wrote:
| One trick I learned is to create a big random file (must be
| random to ensure no funny compression filesystem tricks out
| smart you) called something like
| BIG_DUMMY_DATA_SAFE_TO_DELETE. When you find yourself in a
| catastrophic space situation, you can delete the file and
| have a less panicky recovery process as the immediate
| problem is gone.
| PhilipRoman wrote:
| The funniest consequence of full disk was that I could not
| log in to a server, because the login process required
| writing some tiny temporary file in the user's home. Did not
| find a way to recover it, other than plugging a keyboard in
| it.
| kees99 wrote:
| You were lucky scp uses a binary that typically lives outside
| /bin - in /usr/lib/openssh/ or some such.
|
| Many years ago, I've got to recover a remote server where /usr
| was nuked (and /bin, /sbin, and /lib were all symlinks into
| now-empty /usr). Ended up writing a one-liner perl to convert
| /bin/busybox-static from my local machine into a series of:
| echo -ne "\x7f\x45..." >>~/busybox-static
|
| ...and copy-pasting that, chunk-by-chunk, into a single
| surviving ssh/bash connection, and then used that busybox
| binary to pull in from a backup.
| hinkley wrote:
| I have learned through hard experience that is you ever user
| sudo to edit the sudoers file, create two shell windows
| logged in as root before doing so.
|
| Use visudo to edit the file of course, because not doing so
| can blow everything up by rendering the sudoers file
| unparseable and then everyone is gonna have a bad time.
|
| But also the temptation when altering sudo is to immediately
| log out as super user and try using sudo to do the new
| command. If you've fucked up the file you might not be able
| to sudo anymore. So now use your second shell window to undo
| whatever you just did in the first window.
| ryao wrote:
| I have setup ssh remote forwards to a jump host in the past
| to allow remote access through a firewall. daemontools
| executes scripts exec'ing ssh. ExitOnForwardFailure and
| ServerAliveInterval are set client side with
| ClientAliveInterval and ClientAliveCountMax set server side
| to enable rapid recovery if something goes wrong.
|
| Whenever one of the daemon tools scripts doing remote
| forwards needs to be modified, a second reverse forward
| script is added and the reverse forward from that is used
| for ssh, before changing the original script. The second
| script is removed only after confirming the first still
| works after the edit. This procedure prevents fatfingering
| from locking out remote access, since if something goes
| wrong, you just need to redo the previous step(s) until you
| get things working.
|
| If anyone wants to replicate that, I suggest setting -nNT
| as arguments to ssh and restricting what the user login can
| do via sshd_config.
| malkia wrote:
| What would be safer alternative? pwd # Then
| check something? pushd bin # rm -rf .
|
| Probably still with pitfalls
| 01HNNWZ0MV43FF wrote:
| `rm -rf bin`
| OJFord wrote:
| I think the misguided belief that `./` means 'execute
| script' or something (program that isn't 'installed'?) is
| single-handedly to blame for so much script spaghetti.
| bigstrat2003 wrote:
| And not just for bin. There's probably an edge case where
| you would need to give the ./ prefix to rm, but I've never
| come across it. The vast, vast majority of the time just
| entering the name of the thing is easier _and_ less error-
| prone.
| pphysch wrote:
| Use the full path of the bin dir in your rm rf
| hinkley wrote:
| rm -rf ~/bin could have some nasty consequences if you fat
| finger a return key anywhere in the middle.
|
| Run enough commands enough times and you will find Murphy
| is waiting for you.
|
| If you're just removing a bin directory one time, odds are
| low but not zero. If you're writing a run book for people
| to use, odds are 100% that you will have to help someone
| rebuild at least once.
| cortesoft wrote:
| Write a script that does the steps required?
|
| If the problem is defined enough to create an exact series of
| commands for an operator to execute, it is defined enough to
| create a script to do it for you
| trelane wrote:
| > In hindsight of course we should have written the docs in a
| way that would prevent this exact situation
|
| I would say the bigger failure is relying on a human typing
| things into a terminal rather than automating the tasks or
| changing the system so the task is no longer needed.
| hinkley wrote:
| As if there haven't been outages caused by incorrect
| directory interpolation in scripts.
| trelane wrote:
| Sure, bug happen. They (usually) happen reliably, and tests
| can help prevent/detect them.
|
| It is impossible to test for human errors in advance,
| though.
| micw wrote:
| It's always a good idea to allow sudo to untrained people on
| critical systems...
| hinkley wrote:
| My second boss was a Sun Microsystems enjoyer and he always
| pronounced "superuser" as "stupid user".
|
| After a raised eyebrow he went on to explain, "because when the
| machine is broken it's always because some stupid user did
| something.
|
| It took me a couple more stupiduser incidents of my own before
| I instituted a rule of counting to five before hitting enter on
| any `rm -rf` command.
| Ferret7446 wrote:
| > I instituted a rule of counting to five before hitting
| enter on any `rm -rf` command.
|
| That's just (should be) standard practice. Another risky
| command is `sudo dd`/`sudo cat` for writing disk images,
| always chant the disk device against an fdisk -l listing like
| a magical spell, lest you nuke your main drive.
| rav wrote:
| I often run rm -rf as part of operations (for reasons), and my
| habit is to first run "sudo du -csh" on the paths to be
| deleted, check that the total size makes sense, and then up-
| arrow and replace "du -csh" with "rm -rf".
| mekster wrote:
| Use trash-cli and additionally git commit the target if
| you're nervous before deletion.
| mekster wrote:
| First, you don't let people read and type commands by hand.
| Second, hire a better guy than someone who blindly does sudo
| because the command didn't work.
| nunez wrote:
| Super interesting, though it feels like a response to an
| interview question!
| Dwedit wrote:
| If you deleted /lib, you'd probably be better off reinstalling
| packages while booting off of USB or something. You're gonna have
| downtime because programs won't work correctly.
| LorenDB wrote:
| I also had to wonder why not just liveboot from USB or attach
| the affected boot medium to another system, then use the
| recovery system's fully working tools to just relink the /lib
| folders?
| nullorempty wrote:
| At the start of my career I removed the `x` attribute from all
| files :)
| ivanjermakov wrote:
| chmod -R is too convenient lol
| ryao wrote:
| Of all of the stories I have read here, this is the first to
| make me laugh. Congratulations. :)
| nullorempty wrote:
| ... we had an amazing sysadmin. He had a shell open on that
| box when I came to tell the news. He started to type quickly
| and thoughtfully, trying utilities I haven't even heard of.
| Then, he echoed a small C program that was supposed to set
| `x` on the chmod. Quickly he typed `cc x.c` to compile... and
| then it dawned on him.
| inejge wrote:
| The heroic version of the story is now almost 40 years old[1].
| (One HN mention with the link to a HTMLized version is here[2].)
| In both cases, the upshot is that as long as you have a running
| shell with root privileges, at least one existing executable file
| on the filesystem, and the means to overwrite that file with
| arbitrary binary content, you can write a small program which can
| recreate a skeleton system structure and dig yourself out of the
| hole.
|
| The reason why this keeps happening is that in regular UNIX root
| is omnipotent and the filesystem is ultimately unprotected.
| Immutable systems and restricted execution environments may make
| this a thing of the past.
|
| [1] https://www.wolczko.com/rm.txt [2]
| https://news.ycombinator.com/item?id=7892471
| lloeki wrote:
| A long long time ago the team I was part of managed old unix
| systems.
|
| A coworker telnet'd (or rsh, can't recall) into such a machine to
| do some maintenance and after a while fat fingered:
| umount /
|
| Would you believe it, back then being root meant this was
| absolutely unprotected and the minicomputer OS (some ancient AIX)
| dutifully complied.
|
| The chaos that ensued is but a blur.
| teaearlgraycold wrote:
| Did you just restart?
| WesolyKubeczek wrote:
| Wasn't restarting those AIX dinosaurs a nontrivial thing?
| teaearlgraycold wrote:
| I could believe it. I'm just hoping OP can provide more
| info.
| gleenn wrote:
| One time I was flipping back and forth between directories,
| compiling some code, then checking it, then rm -rf'ing it. I
| accidentally hit up an cd ..'d one too many times. Suddenly the
| rm command hung and I was confused because it should be nearly
| instant as it was a few files. I stared in horror as I was
| accidentally deleting everything in mybhome directory. Luckily
| back then I had a .pr0n directory with a significant amount of
| content. A few things were lost but that .pr0n folder was luckily
| early enough in the list and big enough to slow down the deletion
| of my photos and documents. That's why I always recommend having
| a big "buffer" of video content for such situations, ya know, for
| data integrity ;)
| hinkley wrote:
| Unless you were using tcsh you had to rewrite your shell
| configuration though.
| OKRainbowKid wrote:
| Great idea, I shall see to it right away.
| winwang wrote:
| That's wild and inspirational. Like the scene in Rush Hour when
| a stack of bills saves Tucker from a bullet.
|
| Though I'm moreso tempted to just create a `.1111aaaa-antidumb`
| directory and store my caches and backups there.
|
| This has also un-inspired me from creating a fast `rm`-esque
| utility.
| woleium wrote:
| You could of course alias rm -rf to rm -rf -i
| marc_abonce wrote:
| I always use trash-cli and alias rm to 'echo NO! #'.
|
| Only if the file is too big to fit into the garbage bin, I
| can unalias rm, rm the thing and then reset the alias
| immediately after.
| tarxvf wrote:
| fwiw you can usually bypass aliases ad-hoc.
|
| In Bash I believe instead of `ls` you can `\ls` to get
| the unaliased version.
| contingencies wrote:
| Userpace _pronfs_ provides high latency unlinking?
| mekster wrote:
| Seriously, start using trash-cli. Even Windows from 30 years
| ago had a recycle bin.
|
| I can't grasp how "power users" like Linux users are stuck
| working in primitive environments.
| Ferret7446 wrote:
| If you delete things from a file manager in Linux, they all
| generally go into the Trash too.
|
| And if you rm/del/Remove-Item on Windows, it will also delete
| without sending to the recycle bin.
| o11c wrote:
| > Please note that busybox can't function with a name that is not
| a busybox applet name.
|
| This is somewhat wrong. From ksh/bash/zsh, you can run:
| (exec -a someappname /arbitraryexecutablepath args...)
|
| This won't work on most ash derivatives (including /bin/sh on
| Debian, FreeBSD, or NetBSD), but does work on busybox ash.
|
| The parentheses prevent the `exec` from actually replacing your
| current shell, which might be less important for emergency
| rescues, but which otherwise is often what you want with `exec
| -a`.
| nurple wrote:
| My workstation seems fine: $ ls -R
| /{lib,usr,bin,sbin} ls: cannot access '/sbin': No such file
| or directory /bin: sh /lib: ld-
| linux.so.2 /usr: bin /usr/bin:
| env
|
| Oh right... $ ls -l /usr/bin/env lrwxrwxrwx
| 1 root root 65 Mar 21 23:39 /usr/bin/env ->
| /nix/store/9m68vvhnsq5cpkskphgw84ikl9m6wjwp-coreutils-9.5/bin/env
| $ ldd /usr/bin/env linux-vdso.so.1
| (0x00007ffff7fc4000) libacl.so.1 =>
| /nix/store/dyizbk50iglbibrbwbgw2mhgskwb6ham-
| acl-2.3.2/lib/libacl.so.1 (0x00007ffff7fb3000)
| libattr.so.1 => /nix/store/vlgwyb076hkz7yv96sjnj9msb1jn1ggz-
| attr-2.5.2/lib/libattr.so.1 (0x00007ffff7fab000)
| libgmp.so.10 => /nix/store/dsxb6qvi21bzy21c98kb71wfbdj4lmz7-gmp-
| with-cxx-6.3.0/lib/libgmp.so.10 (0x00007ffff7f06000)
| libc.so.6 => /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.
| 40-66/lib/libc.so.6 (0x00007ffff7d0e000)
| /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib/ld-
| linux-x86-64.so.2 => /nix/store/maxa3xhmxggrc5v2vc0c
| 3pjb79hjlkp9-glibc-2.40-66/lib64/ld-linux-x86-64.so.2
| (0x00007ffff7fc6000)
| MadnessASAP wrote:
| Don't have to worry about trashing FHS if your OS doesn't use
| FHS :-P
| zavec wrote:
| I actually started a short blog series about a similar problem
| where a friend had blown away /bin and a bunch of other stuff,
| but/lib was still there. Unfortunately it didn't end up getting
| anywhere because even though I was able to drop executables on
| the machine with echo and make them executable with a .so from
| lib I wasn't able to get back to root permissions as sudo and
| everything had been blown away and I didn't think I'd have great
| luck trying to find a zero-day in the kernel. It was still a lot
| of fun though.
| Fizzadar wrote:
| I once nuked the entire OS partition on an openvz host. Vz data
| was still good so we ended up copying the root fs from another
| similar box, manually updated the network config and it ran for
| another 4 years until retired.
| throwanem wrote:
| "When you discover yourself to be in a hole, the best first thing
| to do is stop digging."
|
| Some of the best early professional advice I ever received was,
| in moments like these, to keep my hands off the keyboard for at
| least a timed minute.
| ofalkaed wrote:
| Many years ago I wrote a backup script where I did "rm -rf /etc/"
| instead "cp -r /etc/ /mnt/whatever," not sure how I managed that.
| Took ages to figure out what was causing /etc/ to disappear since
| /etc/ going missing often went unnoticed for awhile and I was
| running Arch back in the days when running pacman -syu always
| caused exciting things to happen. I even did a complete backup
| and reinstall trying to figure that one out, and I was extra
| cautious about making a backup of that backup script which I had
| spent so much time on and was fairly proud of since it was my
| first non-trivial bash script.
|
| I also once did "rm -rf /", was deleting a dir which started with
| a "[" and accidentally hit "enter" instead of "\\." That one
| taught me the dangers of absolute paths.
|
| Edit: That last one is not quite right, would not have been an
| absolute path issue, that dir must have ended up in root somehow,
| can't quite remember the details, been too long.
| Vilian wrote:
| Isn't absolute path better practice than relative paths?
| ofalkaed wrote:
| In scripts, yes, but they are not without their dangers and
| can be especially troublesome when working in the terminal
| since almost no one types out and checks the full paths
| constantly, they just let tab completion take care of it and
| assume it worked.
|
| In scripts things like ../../../../file are a pain to read
| and assume everything in the script before all those previous
| dirs worked as it should have and everything is where it
| should be. Cd to an incorrect absolute path produces an error
| code so we can be sure we are in the proper dir, cd ../../../
| will never produce an error and always succeeds even if you
| are at root.
___________________________________________________________________
(page generated 2025-03-22 23:01 UTC)