Cleaning up after zfs send|recv on my zroot / properly transferring installation to new drive reddit / freebsd ---------------------------------------------------------------------- yakketies · 2022-06-27 04:51 So, I recently decided to move my FreeBSD installation onto a new drive. Since I was using ZFS, I achieved this by inserting the new drive, and then using the memstick installer on a USB drive to perform a basic install. After this I dropped to a shell, unmounted all the destination datasets, and then did a ZFS send/recv from the old drive to the new one. After rebooting, I've noticed three annoying issues. 1) **I can no longer log into my normal account or root/toor on the console**. Instead I get amessage to syslog from the login process: This is distinct from entering the wrong password, which just gives me a "Login incorrect" error. After changing my password using passwd (lightdm had no trouble logging me into a graphical session), I can once again login to my normal account, but the root account is still not able to be logged into from the console. 2) **I don't have my usual /dev/ada0pX in /dev**. When I check /dev, I've noticed that for my boot drive, I only have the file for the drive itself, none of the partitions: $ ls /dev/ada* /dev/ada0 Upon further inspection, I see that GEOM does recognise the disk as ada0, but the partitions are using some diskid system exclusively (output truncated):Code: $ geom disk list Geom name: ada0 Providers: 1. Name: ada0 Mediasize: 1000204886016 (932G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e3 descr: WDC WD1003FZEX-00MK2A0 lunid: 50014ee20cf8f5cd ident: WD-WCC3F5ZP44ZU $ geom part list Geom name: diskid/DISK-WD-WCC3F5ZP44ZU modified: false state: OK fwheads: 16 fwsectors: 63 last: 1953525127 first: 40 entries: 128 scheme: GPT Providers: 1. Name: diskid/DISK-WD-WCC3F5ZP44ZUp1 Mediasize: 272629760 (260M) Sectorsize: 512 Mode: r0w0e0 efimedia: HD(1,GPT,043ee012-f2da-11ec-98d8-0cc47a32d38a,0x28,0x82000) label: efiboot0 type: efi index: 1 2. Name: diskid/DISK-WD-WCC3F5ZP44ZUp2 Mediasize: 4294967296 (4.0G) Mode: r0w0e0 efimedia: HD(2,GPT,046d7551-f2da-11ec-98d8-0cc47a32d38a,0x82800,0x800000) label: swap0 type: freebsd-swap index: 2 3. Name: diskid/DISK-WD-WCC3F5ZP44ZUp3 Mediasize: 995635494912 (927G) Mode: r1w1e1 efimedia: HD(3,GPT,0479807d-f2da-11ec-98d8-0cc47a32d38a,0x882800,0x73e84000) label: zfs0 type: freebsd-zfs index: 3 Consumers: 1. Name: diskid/DISK-WD-WCC3F5ZP44ZU Mediasize: 1000204886016 (932G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 Is there any way I can fix this and get back my normal /dev/ada0p1, etc.? 3) **Finally, the bootloader now requires intervention to boot**. At the point where I can drop to an "OK" prompt, I can see that the bootloader tries to boot from the drive's first two partitions before trying what looks like an incomplete ZFS path, "zfs:freebsd-zroot", and failing to find a bootable partition. In order to boot, I have to issue: set currdev=zfs:freebsd-zroot/ROOT/default: load /boot/kernel/kernel load /boot/kernel/opensolaris.ko load /boot/kernel/zfs.ko boot I presume gpart's bootcode option could fix this, but trying to do anything with ada0 just gives me a "no such geom" error, which I'd hazard a guess has something to do with the second issue. Both the drive whose installation I sent over and the new one are 13.1-RELEASE, though the installation has been upgraded from 13.0-RELEASE. I used a 13.1-RELEASE memstick image to do the basic install then the zfs send|recv. Partition type is GPT, and I'm booting using UEFI. The source drive was encrypted with GELI but the destination is not. **Failing all this, however, what is a proper way to transfer a ZFS- based installation of FreeBSD to a new drive?** ---------------------------------------------------------------------- DimestoreProstitute · 2022-06-27 05:29 Look at the "name" entries from your "gpart list", you should see your disk as /dev/diskid/DISK-WD-WCC3F5ZP44ZU. You can get your /dev/ada0 name back by exporting and re-importing your zpool with the appropriate device name, look at the -d flag for zpool-import (EDIT: I think, not at my PC, maybe something like " zpool import -d /dev/ada* poolname" ? ). ---------------------------------------------------------------------- yakketies · 2022-06-27 05:44 Not sure how I'm meant to export the zroot I'm booted into, unfortunately. :( I've also tried to disable GEOM diskid/etc stuff in /boot/loader.conf to no avail. ---------------------------------------------------------------------- DimestoreProstitute · 2022-06-27 05:49 As an alternative, at least for the time being you should be able to substitute /dev/diskid/... for /dev/ada0 for any commands/configs referencing ada0. ---------------------------------------------------------------------- DimestoreProstitute · 2022-06-27 05:52 Otherwise you could try to re-send with the system booted with diskid disabled in loader.conf as you mentioned. There's probably a way to re-import a zroot as well though I'm not familiar with that, perhaps others will chime-in. ---------------------------------------------------------------------- avgapon · 2022-06-27 09:16 We cannot know what went wrong without seeing everything you did for the migration. One thing, you need to set bootfs property of freebsd-zroot to point to freebsd-zroot/ROOT/default. ---------------------------------------------------------------------- usernamefindingsucks · 2022-06-27 14:37 My method would have been to 1. copy the drive partition layout using gpart 2. copy the data from the non-zfs partitions using dd (ignoring the swap partition) 3. add the new swap partition using swapon 4. remove the old swap partition using swapoff 5. add the new zfs partition to the zroot pool as a mirror 6. wait for resilver 7. remove the old drive from the mirror. You can do this whole process without even rebooting. The only caveat I can think of is if you are using encrypted swap, or mirrored swap or both as you have to add some mirroring/encrypting between steps 2 and 3. Some links in case you have mirrored/encrypted swap [https://people.freebsd.org/\~rodrigc/doc/handbook/swap-encrypting.htm l](https://people.freebsd.org/~rodrigc/doc/handbook/swap- encrypting.html) [https://www.freebsd.org/cgi/man.cgi?gmirror(8)](https://www.freebsd.o rg/cgi/man.cgi?gmirror(8)) ---------------------------------------------------------------------- grahamperrin · 2022-06-27 19:08 > … tried to disable GEOM diskid/etc stuff in `/boot/loader.conf` to no avail. How, exactly, did you try? https://old.reddit.com/r/freebsd/comments/vl59td/-/ ---------------------------------------------------------------------- grahamperrin · 2022-06-27 19:16 > … a ZFS send/recv … … > … **a proper way to transfer a ZFS-based installation of FreeBSD to a new drive?** Did you generate a replication stream package? ---------------------------------------------------------------------- yakketies · 2022-06-28 08:31 To the people asking about how I did the transfer, it was more or less: 1. Installed new drive, and booted into a FreeBSD install media. 2. Performed a basic install to get the basic partition structure (EFI, swap, main ZFS partition) going. 3. Once the install had completed, I dropped into a shell, created a recursive snapshot of the root dataset in the old drive's ZFS pool ("moving"), unmounted all datasets on the new drive's ZFS pool, and issued: ​ # zfs send -Rv freebsd-root@moving | zfs recieve freebsd-zroot (old: "freebsd-root", new: "freebsd-zroot") After the send|receive was done, I verified the data had moved, shut down and removed the old drive. Upon booting up, I had to manually massage the bootloader to actually start the new system. ---------------------------------------------------------------------- yakketies · 2022-06-28 08:38 >https://old.reddit.com/r/freebsd/comments/vl59td/-/ I specified `kern.geom.label.disk_ident.enable=0` and `kern.geom.label.gptid.enable=0` in /etc/loader.conf. My guess is the system is reading it too late in the boot process, since the bootloader can't normally read it (is there a way to ensure it is loaded during the "OK" prompt stage?). ---------------------------------------------------------------------- grahamperrin · 2022-06-28 17:19 > `/etc/loader.conf` Instead: `/boot/loader.conf` ---------------------------------------------------------------------- yakketies · 2022-06-29 09:18 Sorry, typo on my post :( I did in fact put it in /boot/loader.conf. ---------------------------------------------------------------------- yakketies · 2022-07-04 09:49 This ended up being the solution. I just created a larger ZFS partition and copied over the contents of the EFI partition, and it worked a treat. No trouble with logging in or geom and the system is bootable right off the bat. ---------------------------------------------------------------------- grahamperrin · 2022-07-04 18:03 > … the solution. … Thanks, you can mark your post: [answered](https://i.imgur.com/pTA9UeG.png)