FreeBSD 13.1p7 with KDE "latest" Desktop kernel crash: How can one dig into the core dump and see the cause? reddit / freebsd ---------------------------------------------------------------------- de_sonnaz · 2023-03-27 09:09 Since a few days my FreeBSD 13.1p7 with KDE "latest" desktop crashed a few times. The first time happened without any change in my system: no upgrades, no hardware change, no USB device inserted, etc. The Desktop is always used with UPS power, I doubt any RAM issues (I checked last time in December, will check again though). **How can one dig into the core dump and see what caused the kernel crash?** # cd /var/crash/ # ls -lt total 10146377 -rw-r--r-- 1 root wheel 226 Mar 27 03:07 core.txt.2 lrwxr-xr-x 1 root wheel 8 Mar 27 03:07 vmcore.last -> vmcore.2 lrwxr-xr-x 1 root wheel 6 Mar 27 03:07 info.last -> info.2 -rw------- 1 root wheel 5681291264 Mar 27 03:07 vmcore.2 -rw------- 1 root wheel 387 Mar 27 03:07 info.2 -rw-r--r-- 1 root wheel 2 Mar 27 03:07 bounds # cat core.txt.2 'version' has unknown type; cast it to its declared type 'version' has unknown type; cast it to its declared type 'version' has unknown type; cast it to its declared type Unable to find matching kernel for /var/crash/vmcore.2 # cat info.2 Dump header from device: /dev/nvd0p3 Architecture: amd64 Architecture Version: 2 Dump Length: 5681291264 Blocksize: 512 Compression: none Dumptime: 2023-03-27 03:06:07 +0200 Hostname: fido.bx.gy Magic: FreeBSD Kernel Dump Version String: FreeBSD 13.1-RELEASE-p6 GENERIC Panic String: general protection fault Dump Parity: 3118795050 Bounds: 2 Dump Status: good ---- **Update:** Many, many thanks to /u/yuripv79 for the patience and all the suggestions. Now I know much more, and in the future I will know what to do. Here is a recap: 1. Check directory /usr/lib/debug and related sub-directories. 2. If they are empty, download proper debug symbols (i.e. for 13.1: http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RELEASE/kernel- dbg.txz) and untar in the proper place: `tar xf kernel-dbg.txz -C /` 3. Run `freebsd-update fetch` and `freebsd-update install` 4. Run the `crashinfo` script using the last kernel dump, i.e. `crashinfo /var/crash/vmcore.last` The script will write debug information as text file, for example, Writing crash summary to /var/crash/core.txt.2. 5. Check the panic part of that text file, it should be small, starting with "panic:" 6. The backtrace should provide at least some hints where it should be reported (zfs: filesystem, etc). 7. If I may add my personal experience: always check the RAM first, using `memtest86` or similar tool. Also, test system without USB devices connected, as a faulty cable or hard disk or device can cause kernel panic on **ANY* system. ---- Thank you again, /u/yuripv79, people like you make the FreeBSD community stands head and shoulder. ---------------------------------------------------------------------- de_sonnaz · 2023-03-27 11:41 From [here](https://freebsdfoundation.org/wp- content/uploads/2022/06/kernal_debugging.pdf): > A kernel dump on its own is not very useful: debuggers require that a core dump be paired with an exact copy of the kernel and its debug info. Since my kernel has no debug symbols, it seems core dumps are not that useful? ---------------------------------------------------------------------- yuripv79 · 2023-03-27 16:16 It also says the following: ``` By default, kernel debug info is split into separate files under /usr/lib/debug. ``` Are the debug symbols not there? Do you have the gdb port/package installed? If yes, what does the following command (taken from the crashinfo script) returns: `gdb -batch -ex 'printf " Version String: %s", version' /boot/kernel/kernel`? ---------------------------------------------------------------------- de_sonnaz · 2023-03-27 16:41 Thank you for helping. `/usr/lib/debug`, and all its sub-directories are empty. Yes, I have `gdb` package installed. Running: `gdb -batch -ex 'printf " Version String: %s", version' /boot/kernel/kernel` I get 'version' has unknown type; cast it to its declared type I am running a stock FreeBSD 13.1. I did not do any change to the kernel. uname -a FreeBSD nx 13.1-RELEASE-p6 FreeBSD 13.1-RELEASE-p6 GENERIC amd64 ---------------------------------------------------------------------- yuripv79 · 2023-03-27 16:56 > /usr/lib/debug, and all its sub-directories are empty. That's unfortunate. You could try fetching the tarball with kernel debug symbols and unpacking it (using `tar xf kernel-dbg.txz -C /`) from http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RELEASE/kernel- dbg.txz. Hopefully, it's close enough for crashinfo to assume it's the same kernel, and if not, not sure if freebsd-update can be run again to update the debug symbols to -p7 version. ---------------------------------------------------------------------- [deleted] · 2023-03-27 17:02 [deleted] ---------------------------------------------------------------------- de_sonnaz · 2023-03-27 17:06 I ran `freebsd-update fetch` and `freebsd-update install` and now I get: Version String: FreeBSD 13.1-RELEASE-p6 GENERIC Running ` kgdb vmcore.last` I get "/var/crash/vmcore.last": not in executable format: file format not recognized ---------------------------------------------------------------------- yuripv79 · 2023-03-27 17:08 That's great! And as everything looks to be intact now, try running the `crashinfo` script on the vmcore file you have, e.g. `crashinfo /var/crash/vmcore.2`. ---------------------------------------------------------------------- de_sonnaz · 2023-03-27 17:24 Thank you deeply for your kind help and patience. I ran the script `crashinfo /var/crash/vmcore.2` I got Writing crash summary to /var/crash/core.txt.2. The core.txt.2 file is a text file, quite big (about 33 Mb). I see some personal data (user name, etc), but it seems not dangerous. Where could I upload it as reference? ---------------------------------------------------------------------- yuripv79 · 2023-03-27 18:37 You could paste the panic part here, that should be small, starting with "panic:" and to the end of stack backtrace ( [270460 – i386 guest panic when booting VirtualBox client (freebsd.org)](https://bugs.freeb sd.org/bugzilla/show_bug.cgi?id=270460#c0) as an example); that should provide at least some hints where it should be reported. ---------------------------------------------------------------------- mirandavanbreukeling · 2023-03-28 06:18 Why not -CURRENT? Build your own stuff and pkg-devel with sudo pkg -f pkg-devel. ---------------------------------------------------------------------- de_sonnaz · 2023-03-28 08:52 Thank you very much. I have put what seems to be the relevant text here: [FreeBSD 13.1 core dumped - part 1](https://old.reddit.com/user/de_son naz/comments/124i9we/freebsd_131_core_dumped_part_1/) [FreeBSD 13.1 core dumped - part 2](https://old.reddit.com/user/de_son naz/comments/124ia33/freebsd_131_core_dumped_part_2/) I am learning a lot here. I will summarise this thread in the OP above, for future use by people in my same (ignorant) situation. Thank you! ---------------------------------------------------------------------- yuripv79 · 2023-03-28 09:11 Apparently it's something wrong in ZFS: ``` #7 0xffffffff80bf3227 at malloc+0x57 #8 0xffffffff8217d5a5 at zfs_acl_node_read+0x225 #9 0xffffffff8217fd86 at zfs_zaccess_aces_check+0x86 #10 0xffffffff8217f512 at zfs_zaccess+0xb2 #11 0xffffffff82312618 at zfs_access+0x68 #12 0xffffffff821904b8 at zfs_freebsd_access+0x48 ``` Try `scrub`ing your pool; if that doesn't help, your best bet is reporting this on freebsd-fs@ and/or freebsd-stable@ mailing lists (make sure to include the "part 1" contents up to "Backtrace stopped"). ---------------------------------------------------------------------- de_sonnaz · 2023-03-28 10:51 Thank you, scrubbing did not report any trouble. I will report these kernel panic to one of the mailing lists you indicated. Before doing that, I will run again `memtest86`, just to be extra sure. ---------------------------------------------------------------------- de_sonnaz · 2023-03-28 11:09 /u/yuripv79, I have updated this original post with all your comments and kind suggestions. ---------------------------------------------------------------------- grahamperrin · 2023-04-02 10:48 Thanks. If you post elsewhere, please add a link from here. I'd like to follow things. ---------------------------------------------------------------------- de_sonnaz · 2023-04-02 15:31 In the meanwhile, I had 2 more kernel panic: following /u/yuripv79 suggestions I looked at the core.txt.n files, and I saw the `KDB: stack backtrace` was different each time: - once related to ZFS (as said earlier) - once to the AMD GPU (this is a Minisforum HX90, with AMD Ryzen 9 5900HX and AMD Radeon graphics card). - once to msdosfs_mount, when I was accessing a sd card via USB card reader. I have tested the RAM again, with no issues. Now I am at a loss where to investigate further. ---------------------------------------------------------------------- grahamperrin · 2023-04-02 17:25 > sudo pkg -f pkg-devel ? ---------------------------------------------------------------------- grahamperrin · 2023-04-02 17:27 Maybe start a new post? Or we can change the opening post from *answered* to *help needed*. ---------------------------------------------------------------------- de_sonnaz · 2023-04-03 05:53 I will start a new post, summarising my findings. Thank You for helping us, here and in so many other places. ---------------------------------------------------------------------- mirandavanbreukeling · 2023-04-15 05:48 Jup, ist’s the Development Package of pkg. you need the -f flag for overwriting the original pkg files. It has a minor update to pkg. ---------------------------------------------------------------------- de_sonnaz · 2023-06-24 05:59 /u/grahamperrin Just to follow up: any test I could plan and execute led nowhere, so much as I believed something was wrong with the hardware. Although, having installed other OS (Windows 10 Enterprise,LTSC, Ubuntu 22.04, OmniOS), I could not replicate any malfunction under any load). Today I saw this, and it struck a chord: https://fosstodon.org/@gabrielesvelto/110592904713090347 The only common trait with the crashes was that Firefox was opened with several hundreds tabs. So, it could be a hidden Firefox bug.