From nobody@FreeBSD.org  Mon Jan  7 08:05:19 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id 715A2B79
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  7 Jan 2013 08:05:19 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 52AE1C7
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  7 Jan 2013 08:05:19 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r0785IkD031202
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 7 Jan 2013 08:05:18 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r0785IeP031201;
	Mon, 7 Jan 2013 08:05:18 GMT
	(envelope-from nobody)
Message-Id: <201301070805.r0785IeP031201@red.freebsd.org>
Date: Mon, 7 Jan 2013 08:05:18 GMT
From: Rasmus Skaarup <freebsd@gal.dk>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Crash: Fatal trap 12: page fault while in kernel mode
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         175091
>Category:       amd64
>Synopsis:       Crash: Fatal trap 12: page fault while in kernel mode
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-amd64
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jan 07 08:10:01 UTC 2013
>Closed-Date:    Tue Mar 05 16:32:58 UTC 2013
>Last-Modified:  Tue Mar 05 16:32:58 UTC 2013
>Originator:     Rasmus Skaarup
>Release:        9.1-RELEASE
>Organization:
>Environment:
FreeBSD thirdhost 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:10 UTC 2012     root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:

On of my virtualized FreeBSD machines has been panic'ing two times within the last two weeks. After the first panic I ran freebsd-update and upgraded to 9.1-RELEASE succesfully. Today the machine panic'ed again.

I have another virtualized FreeBSD machine running on the same host, and it does not exhibit this behaviour.

Here is the output from dmesg, after reboot:

****
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address   = 0x48
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bd5139
stack pointer           = 0x28:0xffffff81625536c0
frame pointer           = 0x28:0xffffff8162553750
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 62083 (httpd)
trap number             = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff80b41133 at vm_fault_hold+0x1b13
#7 0xffffffff80b41cc3 at vm_fault+0x73
#8 0xffffffff80bd84b4 at trap_pfault+0x124
#9 0xffffffff80bd8c6c at trap+0x49c
#10 0xffffffff80bc315f at calltrap+0x8
Uptime: 13h6m22s
*********

This machine is running virtualized on a Centos 6.3 host, with 

qemu-kvm-0.12.1.2-2.295.el6_3.2.x86_64

and the following configuration:

*********
<domain type='kvm'>
  <name>FreeBSD3</name>
  <uuid>77777777-8794-11e1-b2d2-003005fb82e6</uuid>
  <memory unit='KiB'>5242880</memory>
  <currentMemory unit='KiB'>5242880</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/vms/FreeBSD3-SystemDisk.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/vms/FreeBSD3-DataDisk.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0'/>
    <interface type='bridge'>
      <mac address='52:54:00:b2:12:8c'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:b2:12:a2'/>
      <source bridge='br1'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='00:16:8e:1a:f3:a9'/>
      <source bridge='virbr0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.0.0.29' keymap='en-us'>
      <listen type='address' address='10.0.0.29'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>
*******

/boot/loader.conf:
********
virtio_load="YES"
virtio_pci_load="YES"
virtio_blk_load="YES"
if_vtnet_load="YES"
virtio_balloon_load="YES"
*******

pciconf -l
*******
hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x12378086 rev=0x02 hdr=0x00
isab0@pci0:0:1:0:	class=0x060100 card=0x11001af4 chip=0x70008086 rev=0x00 hdr=0x00
atapci0@pci0:0:1:1:	class=0x010180 card=0x11001af4 chip=0x70108086 rev=0x00 hdr=0x00
uhci0@pci0:0:1:2:	class=0x0c0300 card=0x11001af4 chip=0x70208086 rev=0x01 hdr=0x00
none0@pci0:0:1:3:	class=0x068000 card=0x11001af4 chip=0x71138086 rev=0x03 hdr=0x00
vgapci0@pci0:0:2:0:	class=0x030000 card=0x11001af4 chip=0x00b81013 rev=0x00 hdr=0x00
re0@pci0:0:3:0:	class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00
virtio_pci0@pci0:0:4:0:	class=0x050000 card=0x00051af4 chip=0x10021af4 rev=0x00 hdr=0x00
em0@pci0:0:5:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
em1@pci0:0:6:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
virtio_pci1@pci0:0:7:0:	class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00
virtio_pci2@pci0:0:8:0:	class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00
********

>How-To-Repeat:
Happens regularly by itself.
>Fix:


>Release-Note:
>Audit-Trail:

From: John Baldwin <jhb@freebsd.org>
To: freebsd-amd64@freebsd.org
Cc: Rasmus Skaarup <freebsd@gal.dk>,
 freebsd-gnats-submit@freebsd.org
Subject: Re: amd64/175091: Crash: Fatal trap 12: page fault while in kernel mode
Date: Mon, 7 Jan 2013 09:32:36 -0500

 Can you enable crashdumps by setting 'dumpdev="AUTO"' in /etc/rc.conf?
 
 Also, can you run 'gdb /boot/kernel/kernel' and then at the prompt run
 'l *vm_fault_hold+0x1b13' and reply with the output?
 
 -- 
 John Baldwin

From: Rasmus Skaarup <freebsd@gal.dk>
To: John Baldwin <jhb@freebsd.org>
Cc: freebsd-amd64@freebsd.org,
 freebsd-gnats-submit@freebsd.org
Subject: Re: amd64/175091: Crash: Fatal trap 12: page fault while in kernel mode
Date: Mon, 7 Jan 2013 19:11:48 +0100

 Thank you for the quick response. I enabled the setting in rc.conf as =
 you mentioned, and the machine has crashed twice since. The two dumps =
 are uploaded here:
 
 http://gal.dk/crash0.tar.gz
 http://gal.dk/crash1.tar.gz
 
 gdb output for the original error:
 
 (gdb) l *vm_fault_hold+0x1b13
 0xffffffff80b41133 is in vm_fault_hold (/usr/src/sys/vm/vm_fault.c:936).
 931		 * because pmap_enter() may sleep.  We don't put the =
 page
 932		 * back on the active queue until later so that the =
 pageout daemon
 933		 * won't find it (yet).
 934		 */
 935		pmap_enter(fs.map->pmap, vaddr, fault_type, fs.m, prot, =
 wired);
 936		if ((fault_flags & VM_FAULT_CHANGE_WIRING) =3D=3D 0 && =
 wired =3D=3D 0)
 937			vm_fault_prefault(fs.map->pmap, vaddr, =
 fs.entry);
 938		VM_OBJECT_LOCK(fs.object);
 939		vm_page_lock(fs.m);
 940=09
 (gdb)=20
 
 
 The two other crashes, had different excuses. Here is the first:
 
 Fatal trap 9: general protection fault while in kernel mode
 cpuid =3D 3; apic id =3D 03
 instruction pointer     =3D 0x20:0xffffffff81612ace
 stack pointer           =3D 0x28:0xffffff816230a4c0
 frame pointer           =3D 0x28:0xffffff816230a4e0
 code segment            =3D base 0x0, limit 0xfffff, type 0x1b
                         =3D DPL 0, pres 1, long 1, def32 0, gran 1
 processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
 current process         =3D 18778 (imapd)
 processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
 current process         =3D 18778 (imapd)
 trap number             =3D 9
 panic: general protection fault
 cpuid =3D 1
 KDB: stack backtrace:
 #0 0xffffffff809208a6 at kdb_backtrace+0x66
 #1 0xffffffff808ea8be at panic+0x1ce
 #2 0xffffffff80bd8240 at trap_fatal+0x290
 #3 0xffffffff80bd88d5 at trap+0x105
 #4 0xffffffff80bc315f at calltrap+0x8
 #5 0xffffffff8164915d at dnode_free_range+0x29d
 #6 0xffffffff81639d5f at dmu_free_long_range_impl+0x13f
 #7 0xffffffff81639f9c at dmu_free_long_range+0x4c
 #8 0xffffffff816a7839 at zfs_rmnode+0x69
 #9 0xffffffff816be9b6 at zfs_inactive+0x66
 #10 0xffffffff816beb7a at zfs_freebsd_inactive+0x1a
 #11 0xffffffff8097f61d at vinactive+0x8d
 #12 0xffffffff80982de8 at vputx+0x2d8
 #13 0xffffffff80986f4f at kern_unlinkat+0x1df
 #14 0xffffffff80bd7ae6 at amd64_syscall+0x546
 #15 0xffffffff80bc3447 at Xfast_syscall+0xf7
 Uptime: 1h17m3s
 
 (gdb) l *trap_fatal+0x290=20
 0xffffffff80bd8240 is in trap_fatal =
 (/usr/src/sys/amd64/amd64/trap.c:852).
 847			printf("Idle\n");
 848		}
 849=09
 850	#ifdef KDB
 851		if (debugger_on_panic || kdb_active)
 852			if (kdb_trap(type, 0, frame))
 853				return;
 854	#endif
 855		printf("trap number		=3D %d\n", type);
 856		if (type <=3D MAX_TRAP_MSG)
 (gdb)=20
 
 
 The other new crash:
 
 Fatal trap 12: page fault while in kernel mode
 cpuid =3D 1; apic id =3D 01
 fault virtual address   =3D 0x0
 fault code              =3D supervisor read data, page not present
 instruction pointer     =3D 0x20:0xffffffff80bcf1fb
 stack pointer           =3D 0x28:0xffffff8161fef950
 frame pointer           =3D 0x28:0xffffff8161fef990
 code segment            =3D base 0x0, limit 0xfffff, type 0x1b
                         =3D DPL 0, pres 1, long 1, def32 0, gran 1
 processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
 current process         =3D 75787 (httpd)
 trap number             =3D 12
 panic: page fault
 cpuid =3D 1
 KDB: stack backtrace:
 #0 0xffffffff809208a6 at kdb_backtrace+0x66
 #1 0xffffffff808ea8be at panic+0x1ce
 #2 0xffffffff80bd8240 at trap_fatal+0x290
 #3 0xffffffff80bd857d at trap_pfault+0x1ed
 #4 0xffffffff80bd8b9e at trap+0x3ce
 #5 0xffffffff80bc315f at calltrap+0x8
 #6 0xffffffff80bcf290 at pmap_is_modified+0x40
 #7 0xffffffff80b52f7e at vm_page_dontneed+0x17e
 #8 0xffffffff80b4f0cd at vm_object_madvise+0x4dd
 #9 0xffffffff80b49beb at vm_map_madvise+0x1bb
 #10 0xffffffff80b4bff1 at sys_madvise+0x91
 #11 0xffffffff80bd7ae6 at amd64_syscall+0x546
 #12 0xffffffff80bc3447 at Xfast_syscall+0xf7
 Uptime: 5h5m23s
 
 (gdb) l *pmap_is_modified+0x40
 0xffffffff80bcf290 is in pmap_is_modified =
 (/usr/src/sys/amd64/amd64/pmap.c:4264).
 4259		VM_OBJECT_LOCK_ASSERT(m->object, MA_OWNED);
 4260		if ((m->oflags & VPO_BUSY) =3D=3D 0 &&
 4261		    (m->aflags & PGA_WRITEABLE) =3D=3D 0)
 4262			return (FALSE);
 4263		rw_wlock(&pvh_global_lock);
 4264		rv =3D pmap_is_modified_pvh(&m->md) ||
 4265		    ((m->flags & PG_FICTITIOUS) =3D=3D 0 &&
 4266		    =
 pmap_is_modified_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m))));
 4267		rw_wunlock(&pvh_global_lock);
 4268		return (rv);
 (gdb) l *trap_pfault+0x1ed
 0xffffffff80bd857d is in trap_pfault =
 (/usr/src/sys/amd64/amd64/trap.c:773).
 768			if (td->td_intr_nesting_level =3D=3D 0 &&
 769			    PCPU_GET(curpcb)->pcb_onfault !=3D NULL) {
 770				frame->tf_rip =3D =
 (long)PCPU_GET(curpcb)->pcb_onfault;
 771				return (0);
 772			}
 773			trap_fatal(frame, eva);
 774			return (-1);
 775		}
 776=09
 777		return((rv =3D=3D KERN_PROTECTION_FAILURE) ? SIGBUS : =
 SIGSEGV);
 (gdb)=20
 
 (not sure I'm gbd'ing what you need, but let me know).
 
 I am beginning to suspect the hardware, but the strange thing is that =
 the host (CentOS 6.3) and the other virtual machine works completely =
 fine. And the other virtual machine has plenty of user on it.
 
 Best regards,
 Rasmus skaarup
 
From: Rasmus Skaarup <freebsd@gal.dk>
To: "freebsd-gnats-submit@freebsd.org" <FreeBSD-gnats-submit@FreeBSD.org>
Cc:  
Subject: Re: amd64/175091: Crash: Fatal trap 12: page fault while in kernel mode
Date: Mon, 4 Mar 2013 19:53:09 +0100

 --Apple-Mail=_AFD80B73-9089-47C2-B938-E0D0A1CE55A8
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii
 
 I have opened at new bug instead of this one, with more crashes. 
 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=176636
 
 Br
 Rasmus
 
 --Apple-Mail=_AFD80B73-9089-47C2-B938-E0D0A1CE55A8--
State-Changed-From-To: open->closed 
State-Changed-By: linimon 
State-Changed-When: Tue Mar 5 16:31:26 UTC 2013 
State-Changed-Why:  
see kern/176636. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175091 
>Unformatted:
