From root@nook.more.net  Mon Jun 18 15:11:01 2007
Return-Path: <root@nook.more.net>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 9A61C16A421
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 18 Jun 2007 15:11:01 +0000 (UTC)
	(envelope-from root@nook.more.net)
Received: from nook.more.net (nook.more.net [207.160.130.11])
	by mx1.freebsd.org (Postfix) with ESMTP id 5F5AB13C44B
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 18 Jun 2007 15:11:01 +0000 (UTC)
	(envelope-from root@nook.more.net)
Received: from localhost (localhost.more.net [127.0.0.1])
	by nook.more.net (Postfix) with ESMTP id D0DE6C38FE
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 18 Jun 2007 09:55:31 -0500 (CDT)
Received: from nook.more.net ([127.0.0.1])
	by localhost (nook.more.net [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id gmrlXqTpSj9p for <FreeBSD-gnats-submit@freebsd.org>;
	Mon, 18 Jun 2007 09:55:31 -0500 (CDT)
Received: by nook.more.net (Postfix, from userid 0)
	id 6E222C38FA; Mon, 18 Jun 2007 09:55:31 -0500 (CDT)
Message-Id: <20070618145531.6E222C38FA@nook.more.net>
Date: Mon, 18 Jun 2007 09:55:31 -0500 (CDT)
From: dan@more.net
Reply-To: dan@more.net
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Fatal trap 12: page fault while in kernel mode
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         113823
>Category:       kern
>Synopsis:       [panic] Fatal trap 12: page fault while in kernel mode
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jun 18 15:20:01 GMT 2007
>Closed-Date:    Thu Sep 27 20:51:54 GMT 2007
>Last-Modified:  Thu Sep 27 20:51:54 GMT 2007
>Originator:     Dan D Niles
>Release:        FreeBSD 6.2-RELEASE-p5 i386
>Organization:
MOREnet - Missouri Research and Education Network
>Environment:
System: FreeBSD <hostname> 6.2-RELEASE-p5 FreeBSD 6.2-RELEASE-p5 #0: Tue Jun 12 13:39:25 CDT 2007 root@<hostname>:/usr/obj/usr/src/sys/SMP i386

Server is a Dell 2650 running postfix, mysql, amavisd-new, and clamav.
>Description:
# kgdb kernel.debug /var/crash/vmcore.0 
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address   = 0x104
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc066c761
stack pointer           = 0x28:0xe4f8ec90
frame pointer           = 0x28:0xe4f8ec9c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = resume, IOPL = 0
current process         = 5 (thread taskq)
trap number             = 12
panic: page fault
cpuid = 1
Uptime: 1d6h4m40s
Dumping 2047 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 2047MB (524016 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15

#0  doadump () at pcpu.h:165
165             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc067553a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0675861 in panic (fmt=0xc08e4721 "%s")
    at /usr/src/sys/kern/kern_shutdown.c:565
#3  0xc088e2dc in trap_fatal (frame=0xe4f8ec50, eva=260)
    at /usr/src/sys/i386/i386/trap.c:837
#4  0xc088da4e in trap (frame=
      {tf_fs = -969015288, tf_es = -942342104, tf_ds = -453509080, tf_edi = -969003008, tf_esi = 4, tf_ebp = -453448548, tf_isp = -453448580, tf_ebx = -955327240, tf_edx = 6, tf_ecx = 0, tf_eax = 1, tf_trapno = 12, tf_err = 0, tf_eip = -1067006111, tf_cs = 32, tf_eflags = 65538, tf_esp = -955736776, tf_ss = 4})
    at /usr/src/sys/i386/i386/trap.c:270
#5  0xc0879d8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6  0xc066c761 in _mtx_lock_sleep (m=0xc70edcf8, tid=3325964288, opts=0, 
    file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:546
#7  0xc06bbac6 in unp_gc (arg=0x0, pending=2)
    at /usr/src/sys/kern/uipc_usrreq.c:1714
#8  0xc06961e3 in taskqueue_run (queue=0xc64b0200)
    at /usr/src/sys/kern/subr_taskqueue.c:257
#9  0xc06966c6 in taskqueue_thread_loop (arg=0x1)
    at /usr/src/sys/kern/subr_taskqueue.c:376
#10 0xc065ec7d in fork_exit (callout=0xc0696634 <taskqueue_thread_loop>, 
    arg=0xc09e3ca8, frame=0xe4f8ed38) at /usr/src/sys/kern/kern_fork.c:821
#11 0xc0879dec in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208
	
>How-To-Repeat:
	I can reproduce within a couple days by doubling the workload
on the box.
>Fix:

>Release-Note:
>Audit-Trail:

From: Kenneth Vestergaard Schmidt <kvs@pil.dk>
To: bug-followup@FreeBSD.org, dan@more.net
Cc:  
Subject: Re: kern/113823: [panic] Fatal trap 12: page fault while in kernel mode
Date: Tue, 26 Jun 2007 11:31:21 +0200

 Hello.
 
 We're experiencing the same problems, too, on a lot of our Sun Fire
 x2100 (AMD64) servers.
 
 We've experienced it on web-servers (mostly just Apache and postfix),
 and machines with 5-10 jails, running Apache, MySQL, postfix, and the
 likes.
 
 The problem has appeared on everything from 6.2-RELEASE to 6.2-p5.
 
 machine amd64
 cpu HAMMER
 options SMP
 options COMPAT_IA32
 
 using em(4), gmirror and pf(4).
 
 It seems kern/111458 is related, too.
 
 We have boxes to spare, if anything needs testing, but I haven't found a
 way to reproduce it yet, besides just waiting. It might be related to
 load, or number of processes, but I have no hard evidence.
 
 
 Unread portion of the kernel message buffer:
 
 frame pointer           =3D 0x10:0x4
 code segment            =3D base 0x0, limit 0xfffff, type 0x1b
                         =3D DPL 0, pres 1, long 1, def32 0, gran 1
 processor eflags        =3D resume, IOPL =3D 0
 current process         =3D 9 (thread taskq)
 trap number             =3D 12
 panic: page fault
 cpuid =3D 1
 Uptime: 24d7h45m5s
 Physical memory: 504 MB
 Dumping 154 MB: 139 123 107 91 75 59 43 27 11
 
 #0  doadump () at pcpu.h:172
 172     pcpu.h: No such file or directory.
         in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:172
 #1  0x0000000000000004 in ?? ()
 #2  0xffffffff80280b77 in boot (howto=3D260) at /usr/dana/src/freebsd6/src/=
 sys/kern/kern_shutdown.c:409
 #3  0xffffffff80281211 in panic (fmt=3D0xffffff001ed41980 "\b=AA=D5\036")
     at /usr/dana/src/freebsd6/src/sys/kern/kern_shutdown.c:565
 #4  0xffffffff803cb6ef in trap_fatal (frame=3D0xffffff001ed41980, eva=3D184=
 46742974715243016)
     at /usr/dana/src/freebsd6/src/sys/amd64/amd64/trap.c:660
 #5  0xffffffff803cbc16 in trap (frame=3D
       {tf_rdi =3D 19, tf_rsi =3D -1098994411136, tf_rdx =3D 6, tf_rcx =3D 3=
 221225730, tf_r8 =3D -1796719344, tf_r9 =3D -1099004634760, tf_rax =3D 1, t=
 f_rbx =3D -1099413515464, tf_rbp =3D 4, tf_r10 =3D -2141530056, tf_r11 =3D =
 0, tf_r12 =3D -1098994411136, tf_r13 =3D 4, tf_r14 =3D 1, tf_r15 =3D 20, tf=
 _trapno =3D 12, tf_addr =3D 396, tf_flags =3D -1099413515464, tf_err =3D 0,=
  tf_rip =3D -2144901513, tf_cs =3D 8, tf_rflags =3D 65538, tf_rsp =3D -1796=
 719760, tf_ss =3D 16}) at /usr/dana/src/freebsd6/src/sys/amd64/amd64/trap.c=
 :238
 #6  0xffffffff803b6ebb in calltrap () at /usr/dana/src/freebsd6/src/sys/amd=
 64/amd64/exception.S:168
 #7  0xffffffff80276677 in _mtx_lock_sleep (m=3D0xffffff0005d91338, tid=3D18=
 446742974715140480, opts=3D6,=20
     file=3D0xc0000102 <Address 0xc0000102 out of bounds>, line=3D-179671934=
 4)
     at /usr/dana/src/freebsd6/src/sys/kern/kern_mutex.c:546
 #8  0xffffffff802d589d in unp_gc (arg=3D0x13, pending=3D517216640)
     at /usr/dana/src/freebsd6/src/sys/kern/uipc_usrreq.c:1714
 #9  0xffffffff802a8045 in taskqueue_run (queue=3D0xffffff00007d5c00)
     at /usr/dana/src/freebsd6/src/sys/kern/subr_taskqueue.c:257
 #10 0xffffffff802a8d95 in taskqueue_thread_loop (arg=3D0x13)
     at /usr/dana/src/freebsd6/src/sys/kern/subr_taskqueue.c:376
 #11 0xffffffff80267bf7 in fork_exit (callout=3D0xffffffff802a8d10 <taskqueu=
 e_thread_loop>,=20
     arg=3D0xffffffff805b4e70, frame=3D0xffffffff94e83c50)
     at /usr/dana/src/freebsd6/src/sys/kern/kern_fork.c:821
 #12 0xffffffff803b721e in fork_trampoline ()
     at /usr/dana/src/freebsd6/src/sys/amd64/amd64/exception.S:394
 
 --=20
 Kenneth Schmidt
 pil.dk

From: Kai Storbeck <kai@xs4all.nl>
To: bug-followup@FreeBSD.org, dan@more.net
Cc:  
Subject: Re: kern/113823: [panic] Fatal trap 12: page fault while in kernel
 mode
Date: Thu, 05 Jul 2007 14:57:04 +0200

 Hi Dan,
 
 With some fancy searching I found your report in the PR database; we're 
 getting the exact same backtrace from our Dovecot mailservers (serving 
 IMAP) with an NFS backend. Backtrace is pasted below.
 
 Did you get any progression on this bug? Can I help? We're experiencing 
 it a few times per week on 20 imapservers, so not that often, but we 
 could use a fix :)
 
 
 Best regards,
 Kai
 
 Fatal trap 12: page fault while in kernel mode
 cpuid = 2; apic id = 06
 fault virtual address   = 0x104E
 fault code              = supervisor read, page not presentx
 
 #0  doadump () at pcpu.h:165
 165     pcpu.h: No such file or directory.
          in pcpu.h
 
 (kgdb) bt
 #0  doadump () at pcpu.h:165
 #1  0xc0670918 in boot (howto=260) at ../../../kern/kern_shutdown.c:409
 #2  0xc0670bfa in panic (fmt=0xc08d0a0d "%s")
      at ../../../kern/kern_shutdown.c:565
 #3  0xc087819c in trap_fatal (frame=0xe8916c30, eva=260)
      at ../../../i386/i386/trap.c:837
 #4  0xc087794a in trap (frame=
        {tf_fs = -393150456, tf_es = -1064959960, tf_ds = -393150424, 
 tf_edi = -935090688, tf_esi = -900488032, tf_ebp = -393122692, tf_isp = 
 -393122724, tf_ebx = 4, tf_edx = 6, tf_ecx = 2, tf_eax = 1, tf_trapno = 
 12, tf_err = 0, tf_eip = -1067020483, tf_cs = 32, tf_eflags = 65538, 
 tf_esp = 1714, tf_ss = -1064340051})
      at ../../../i386/i386/trap.c:270
 #5  0xc08649ea in calltrap () at ../../../i386/i386/exception.s:139
 #6  0xc0668f3d in _mtx_lock_sleep (m=0xca53a4a0, tid=3359876608, opts=0,
      file=0xc08f75ad "../../../kern/uipc_usrreq.c", line=1714)
      at ../../../kern/kern_mutex.c:546
 #7  0xc0668b93 in _mtx_lock_flags (m=0x2, opts=0,
      file=0xc08f75ad "../../../kern/uipc_usrreq.c", line=1714)
      at ../../../kern/kern_mutex.c:288
 #8  0xc06b204b in unp_gc (arg=0x0, pending=1)
      at ../../../kern/uipc_usrreq.c:1714
 #9  0xc068f7c0 in taskqueue_run (queue=0xc843ca80)
      at ../../../kern/subr_taskqueue.c:257
 #10 0xc068fb3e in taskqueue_thread_loop (arg=0x1)
      at ../../../kern/subr_taskqueue.c:376
 #11 0xc065d184 in fork_exit (callout=0xc068faf4 <taskqueue_thread_loop>,
      arg=0xc09df4e8, frame=0xe8916d38) at ../../../kern/kern_fork.c:821
 #12 0xc0864a4c in fork_trampoline () at ../../../i386/i386/exception.s:208
 
 
 -- 
 This was an above the .signature production

From: Dan D Niles <dan@more.net>
To: bug-followup@FreeBSD.org, dan@more.net
Cc:  
Subject: Re: kern/113823: [panic] Fatal trap 12: page fault while in kernel
	mode
Date: Thu, 05 Jul 2007 16:03:02 -0500

 I've swapped out all the hardware, including the OS disk.  It is still
 panicking within 24 hours.
 
 I'm going to downgrade to FreeBSD 6.1 shortly, I need this server
 working.
 
 Is there any additional information I can provide from the vmcore?
 
 Is there any additional debugging I should do?
 
 Thanks,
 
 Dan
 
 

From: John Baldwin <jhb@FreeBSD.org>
To: bug-followup@FreeBSD.org, dan@more.net
Cc:  
Subject: Re: kern/113823: [panic] Fatal trap 12: page fault while in kernel mode
Date: Wed, 25 Jul 2007 12:40:34 -0400

 This is fixed in 6.2-stable and has been in re@'s queue for an errata fix for 
 RELENG_6_2 since the release was made.  Here is the relevant commit:
 
 jhb         2007-01-12 16:21:29 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_6)
     sys/kern             kern_descrip.c uipc_usrreq.c 
     sys/sys              file.h 
   Log:
   MFC: Close a race between UNIX domain pcb garbage collection (unp_gc()) and
   file descriptor teardown (fdrop()) by adding a new garbage collection flag
   FWAIT.
   
   Revision    Changes    Path
   1.279.2.10  +11 -0     src/sys/kern/kern_descrip.c
   1.155.2.7   +27 -9     src/sys/kern/uipc_usrreq.c
   1.70.2.2    +1 -0      src/sys/sys/file.h
 
 -- 
 John Baldwin
State-Changed-From-To: open->patched 
State-Changed-By: rodrigc 
State-Changed-When: Thu Jul 26 01:58:52 UTC 2007 
State-Changed-Why:  
Fixed in RELENG_6 branch. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=113823 
State-Changed-From-To: patched->closed 
State-Changed-By: jhb 
State-Changed-When: Thu Sep 27 20:51:06 UTC 2007 
State-Changed-Why:  
This is a duplicate of kern/107325. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=113823 
>Unformatted:
