From nobody@FreeBSD.org  Thu Oct 17 04:17:34 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTP id 8A0699CF
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 17 Oct 2013 04:17:34 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from oldred.freebsd.org (oldred.freebsd.org [8.8.178.121])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id 5DC8E2FBB
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 17 Oct 2013 04:17:34 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id r9H4HYBn048214
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 17 Oct 2013 04:17:34 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id r9H4HYnZ048213;
	Thu, 17 Oct 2013 04:17:34 GMT
	(envelope-from nobody)
Message-Id: <201310170417.r9H4HYnZ048213@oldred.freebsd.org>
Date: Thu, 17 Oct 2013 04:17:34 GMT
From: Shawn Wallbridge <shawn.wallbridge@imaginaryforces.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Machine kernel panic'ed, now fsck is segfaulting
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         183042
>Category:       kern
>Synopsis:       [panic] Machine kernel panic'ed, now fsck is segfaulting
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Oct 17 04:20:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Sun Apr 20 02:03:07 UTC 2014
>Originator:     Shawn Wallbridge
>Release:        9.2-RELEASE
>Organization:
Imaginary Forces
>Environment:
FreeBSD mercury 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013     root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
A few panics on the box over the last month, did an upgrade to 9.2 last week. Tonight it panic'ed again, after it came back up, it couldn't background fsck one of the file systems. I brought it back up by commenting that filesystem out.

Now I am trying to do an fsck on the filesystem, but it segfaults.

root@mercury:/var/log # fsck /dev/da0p1
fsck: Could not determine filesystem type

root@mercury:/var/log # fsck -t ufs /dev/da0p1
** /dev/da0p1
fsck: /dev/da0p1: Segmentation fault: 11

It's running 9.2-RELEASE

uname -a
FreeBSD mercury 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26
22:50:31 UTC 2013root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC
amd64

The filesystem is on a LSI RAID Card, 9750-24i4e. The controller says
everything is fine.

I tried looking at the core file, but it doesn't mean anything to me.

# gdb core fsck_ufs.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...core: No such file or
directory.

Core was generated by `fsck_ufs'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000405cce in ?? ()
(gdb) bt
#0 0x0000000000405cce in ?? ()
#1 0xffffffff90000000 in ?? ()
#2 0x000000000040ef6b in ?? ()
#3 0x0000000000624e10 in ?? ()
#4 0x000000080061f8cb in ?? ()
#5 0x0000007b7100ff00 in ?? ()
#6 0x00000000000121a0 in ?? ()
#7 0x0000007b00000005 in ?? ()
#8 0x00000000525f4c3b in ?? ()
#9 0x00000000021036a8 in ?? ()
#10 0x00000000525f4c3b in ?? ()
#11 0x00000000021036a8 in ?? ()
#12 0x00000000525f4c3b in ?? ()
#13 0x00000000021036a8 in ?? ()
#14 0x0000000000000000 in ?? ()
(gdb) quit


>How-To-Repeat:
Just run fsck again..

 fsck_ufs -y /dev/da0p1
** /dev/da0p1
Segmentation fault (core dumped)
>Fix:


>Release-Note:
>Audit-Trail:

From: Shawn Wallbridge <shawn.wallbridge@imaginaryforces.com>
To: "bug-followup@FreeBSD.org" <bug-followup@FreeBSD.org>, Shawn Wallbridge
	<shawn.wallbridge@imaginaryforces.com>
Cc:  
Subject: Re: misc/183042: Machine kernel panic&#39;ed, now fsck is
 segfaulting
Date: Thu, 17 Oct 2013 12:32:37 +0000

 I was able to grab the fsck_ufs binary from a 9.0-RELEASE box and run it,
 which worked. I was able to clean the filesystem and re-mount it.
 
 Even _AFTER_ the filesystem was clean, the 9.2-RELEASE binary will
 segfault.
 
 Here is the truss output from running the 9.2-RELEASE binary on the clean
 (unmounted) filesystem.
 
 root@mercury:~ # fsck /dev/da0p1
 ** /dev/da0p1
 fsck: /dev/da0p1: Segmentation fault: 11
 root@mercury:~ # truss fsck /dev/da0p1
 mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D
 34366144512 (0x80061c000)
 issetugid(0x80081ca60,0x7fffffffefd5,0x40,0x0,0xffff80080081da8b,0x0) =3D 0
 (0x0)
 lstat("/etc",{ mode=3Ddrwxr-xr-x ,inode=3D15730176,size=3D2048,blksize=3D32=
 768 })
 =3D 0 (0x0)
 lstat("/etc/libmap.conf",{ mode=3D-rw-r--r--
 ,inode=3D15730599,size=3D111,blksize=3D32768 }) =3D 0 (0x0)
 open("/etc/libmap.conf",O_RDONLY,040345040)      =3D 3 (0x3)
 fstat(3,{ mode=3D-rw-r--r-- ,inode=3D15730599,size=3D111,blksize=3D32768 })=
  =3D 0
 (0x0)
 mmap(0x0,111,PROT_READ,MAP_PRIVATE,3,0x0)        =3D 34366177280 (0x8006240=
 00)
 close(3)                                         =3D 0 (0x0)
 lstat("/usr",{ mode=3Ddrwxr-xr-x ,inode=3D16773504,size=3D512,blksize=3D327=
 68 }) =3D
 0 (0x0)
 lstat("/usr/local",{ mode=3Ddrwxr-xr-x
 ,inode=3D16773512,size=3D512,blksize=3D32768 }) =3D 0 (0x0)
 lstat("/usr/local/etc",{ mode=3Ddrwxr-xr-x
 ,inode=3D17740802,size=3D1024,blksize=3D32768 }) =3D 0 (0x0)
 lstat("/usr/local/etc/libmap.d",0x7fffffffb8b0)  ERR#2 'No such file or
 directory'
 munmap(0x800624000,111)                          =3D 0 (0x0)
 open("/var/run/ld-elf.so.hints",O_RDONLY,057)    =3D 3 (0x3)
 read(3,"Ehnt\^A\0\0\0\M^@\0\0\0-\0\0\0\0"...,128) =3D 128 (0x80)
 lseek(3,0x80,SEEK_SET)                           =3D 128 (0x80)
 read(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,45) =3D 45 (0x2d)
 close(3)                                         =3D 0 (0x0)
 access("/lib/libc.so.7",0)                       =3D 0 (0x0)
 open("/lib/libc.so.7",O_RDONLY,040352740)        =3D 3 (0x3)
 fstat(3,{ mode=3D-r--r--r-- ,inode=3D46548547,size=3D1406888,blksize=3D3276=
 8 }) =3D
 0 (0x0)
 mmap(0x0,4096,PROT_READ,MAP_PRIVATE|0x40000,3,0x0) =3D 34366177280
 (0x800624000)
 mmap(0x0,3518464,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) =3D
 34368249856 (0x80081e000)
 mmap(0x80081e000,1269760,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCO
 RE|0x40000,3,0x0) =3D 34368249856 (0x80081e000)
 mmap(0x800b54000,45056,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|0x40000,3
 ,0x136000) =3D 34371616768 (0x800b54000)
 mmap(0x800b5f000,106496,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON
 ,-1,0x0) =3D 34371661824 (0x800b5f000)
 munmap(0x800624000,4096)                         =3D 0 (0x0)
 close(3)                                         =3D 0 (0x0)
 munmap(0x800623000,4096)                         =3D 0 (0x0)
 mmap(0x0,102400,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D
 34366173184 (0x800623000)
 sysarch(0x81,0x7fffffffd2a0,0x8006210c8,0x0,0xffffffffffaca580,0x8080808080
 808080) =3D 0 (0x0)
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM
 |SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXF
 SZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)                 =3D 0 (0x0)
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM
 |SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXF
 SZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)                 =3D 0 (0x0)
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM
 |SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXF
 SZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)                 =3D 0 (0x0)
 stat("/dev/da0p1",{ mode=3Dcrw-r----- ,inode=3D123,size=3D0,blksize=3D4096 =
 }) =3D 0
 (0x0)
 getfsstat(0x0,0x0,0x2,0x2f,0x3d,0x7fffffffef7d)  =3D 2 (0x2)
 readlink("/etc/malloc.conf",0x7fffffffcea0,1024) ERR#2 'No such file or
 directory'
 issetugid(0x80092da49,0x7fffffffcea0,0xffffffffffffffff,0x0,0x2,0x7fffffffe
 f7d) =3D 0 (0x0)
 break(0x800000)                                  =3D 0 (0x0)
 mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D
 34371768320 (0x800b79000)
 mmap(0x800f79000,552960,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0)
 =3D 34375962624 (0x800f79000)
 munmap(0x800b79000,552960)                       =3D 0 (0x0)
 getfsstat(0x800c07400,0x588,0x2,0x8,0x605630,0x7fffffffc9e0) =3D 2 (0x2)
 stat("/dev/raid/r0p2",{ mode=3Dcrw-r----- ,inode=3D135,size=3D0,blksize=3D4=
 096 })
 =3D 0 (0x0)
 stat("/dev/devfs",0x7fffffffd3b0)                ERR#2 'No such file or dir=
 ectory'
 issetugid(0x0,0x7fffffffd3b0,0x0,0x0,0x101010101010101,0x8080808080808080)
 =3D 0 (0x0)
 open("/etc/fstab",O_CLOEXEC,02)                  =3D 3 (0x3)
 fcntl(3,F_GETFL,)                                =3D 0 (0x0)
 fstat(3,{ mode=3D-rw-r--r-- ,inode=3D15730545,size=3D136,blksize=3D32768 })=
  =3D 0
 (0x0)
 read(3,"# Device\tMountpoint\tFStype\tOp"...,32768) =3D 136 (0x88)
 statfs(0x800932c34,0x7fffffffcee0,0x800b611af,0x0,0x9,0x800b611b1) =3D 0
 (0x0)
 stat("/dev/raid/r0p2",{ mode=3Dcrw-r----- ,inode=3D135,size=3D0,blksize=3D4=
 096 })
 =3D 0 (0x0)
 read(3,0x800c1f000,32768)                        =3D 0 (0x0)
 lseek(3,0x0,SEEK_CUR)                            =3D 136 (0x88)
 lseek(3,0x0,SEEK_SET)                            =3D 0 (0x0)
 read(3,"# Device\tMountpoint\tFStype\tOp"...,32768) =3D 136 (0x88)
 statfs(0x800932c34,0x7fffffffcee0,0x800b611af,0x0,0x9,0x800b611b1) =3D 0
 (0x0)
 stat("/dev/raid/r0p2",{ mode=3Dcrw-r----- ,inode=3D135,size=3D0,blksize=3D4=
 096 })
 =3D 0 (0x0)
 vfork(0x7fffffffd108,0x4019ba,0x8,0x800c79200,0x0,0x0) =3D 1847 (0x737)
 ** /dev/da0p1
 wait4(0x737,0x7fffffffd11c,0x0,0x0,0x7fffffffc7a0,0xffffffff) =3D 1847
 (0x737)
 stat("/usr/share/nls/C/libc.cat",0x7fffffffcbb0) ERR#2 'No such file or
 directory'
 stat("/usr/share/nls/libc/C",0x7fffffffcbb0)     ERR#2 'No such file or
 directory'
 stat("/usr/local/share/nls/C/libc.cat",0x7fffffffcbb0) ERR#2 'No such file
 or directory'
 stat("/usr/local/share/nls/libc/C",0x7fffffffcbb0) ERR#2 'No such file or
 directory'
 fsck: write(2,"fsck: ",6)                                =3D 6 (0x6)
 /dev/da0p1: Segmentation fault: 11write(2,"/dev/da0p1: Segmentation fault:
 "...,34) =3D 34 (0x22)
 
 write(2,"\n",1)                                  =3D 1 (0x1)
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM
 |SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXF
 SZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)                 =3D 0 (0x0)
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM
 |SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXF
 SZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)                 =3D 0 (0x0)
 sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM
 |SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXF
 SZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) =3D 0 (0x0)
 sigprocmask(SIG_SETMASK,0x0,0x0)                 =3D 0 (0x0)
 process exit, rval =3D 1
 
 
 ________________________________
 
 This e-mail is intended only for the named person or entity to which it is =
 addressed and contains valuable business information that is proprietary, p=
 rivileged, confidential and/or otherwise protected from disclosure. If you =
 received this e-mail in error, any review, use, dissemination, distribution=
  or copying of this e-mail is strictly prohibited. Please notify us immedia=
 tely of the error via e-mail to <ifpostmaster> postmaster@imaginaryforces.c=
 om and please delete the e-mail from your system, retaining no copies in an=
 y media. We appreciate your cooperation.
 
 ...imaginaryforces.com...=0D
 
>Unformatted:
