From marius@newtrinity.zeist.de  Sun Oct  5 11:30:38 2003
Return-Path: <marius@newtrinity.zeist.de>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id E293316A4B3
	for <FreeBSD-gnats-submit@freebsd.org>; Sun,  5 Oct 2003 11:30:38 -0700 (PDT)
Received: from newtrinity.zeist.de (newtrinity.zeist.de [217.24.217.8])
	by mx1.FreeBSD.org (Postfix) with ESMTP id BB08A43FF5
	for <FreeBSD-gnats-submit@freebsd.org>; Sun,  5 Oct 2003 11:30:37 -0700 (PDT)
	(envelope-from marius@newtrinity.zeist.de)
Received: from newtrinity.zeist.de (localhost [127.0.0.1])
	by newtrinity.zeist.de (8.12.10/8.12.10/ZEIST.DE) with ESMTP id h95IUaYq090683;
	Sun, 5 Oct 2003 20:30:36 +0200 (CEST)
	(envelope-from marius@newtrinity.zeist.de)
Received: (from marius@localhost)
	by newtrinity.zeist.de (8.12.10/8.12.10/Submit) id h95IUV1b090682;
	Sun, 5 Oct 2003 20:30:31 +0200 (CEST)
	(envelope-from marius)
Message-Id: <200310051830.h95IUV1b090682@newtrinity.zeist.de>
Date: Sun, 5 Oct 2003 20:30:31 +0200 (CEST)
From: Marius Strobl <marius@alchemy.franken.de>
Reply-To: Marius Strobl <marius@alchemy.franken.de>
To: FreeBSD-gnats-submit@freebsd.org
Cc: Bruce M Simpson <bms@spc.org>
Subject: mlockall(2) related VM-bug
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         57611
>Category:       kern
>Synopsis:       mlockall(2) related VM-bug
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    bms
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Oct 05 11:40:15 PDT 2003
>Closed-Date:    Thu Oct 16 12:27:29 PDT 2003
>Last-Modified:  Thu Oct 16 12:27:29 PDT 2003
>Originator:     Marius Strobl
>Release:        FreeBSD 5.1-CURRENT i386
>Organization:
>Environment:
>Description:
Since Bruce M Simpson commited the support for the mlockall(2) and
munlockall(2) system calls on Aug 11 07:14:07 2003 UTC a cdrecord compiled
on FreeBSD since then causes the kernel to panic.
Backtrace of the panic I get with FreeBSD sources as of today:

GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic: mutex vm object not owned at /usr/src/sys/vm/vm_page.c:762
panic messages:
---
panic: mutex vm object not owned at /usr/src/sys/vm/vm_page.c:762
cpuid = 0; lapic.id = 01000000
panic: from debugger
cpuid = 0; lapic.id = 01000000
boot() called on cpu#0
Uptime: 2m11s
Dumping 512 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496
---
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
240             dumping++;
(kgdb) where
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
#1  0xc050c3a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:372
#2  0xc050c788 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
#3  0xc043eff2 in db_panic () at /usr/src/sys/ddb/db_command.c:450
#4  0xc043ef6a in db_command (last_cmdp=0xc06d8e00, cmd_table=0x0, 
    aux_cmd_tablep=0xc0693a6c, aux_cmd_tablep_end=0xc0693a70)
    at /usr/src/sys/ddb/db_command.c:346
#5  0xc043f078 in db_command_loop () at /usr/src/sys/ddb/db_command.c:472
#6  0xc0441db9 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_trap.c:73
#7  0xc0623143 in kdb_trap (type=3, code=0, regs=0xddb95ae8)
    at /usr/src/sys/i386/i386/db_interface.c:171
#8  0xc063b5ce in trap (frame=
      {tf_fs = 24, tf_es = 16, tf_ds = 16, tf_edi = -1066954496, tf_esi = 1, tf_ebp = -575055052, tf_isp = -575055084, tf_ebx = 0, tf_edx = 0, tf_ecx = 0, tf_eax = 18, tf_trapno = 3, tf_err = 0, tf_eip = -1067305938, tf_cs = 8, tf_eflags = 642, tf_esp = -1066874695, tf_ss = -1066952000})
    at /usr/src/sys/i386/i386/trap.c:578
#9  0xc0624a48 in calltrap () at {standard input}:103
#10 0xc050c721 in panic (fmt=0xc0679100 "mutex %s not owned at %s:%d")
    at /usr/src/sys/kern/kern_shutdown.c:534
#11 0xc050352f in _mtx_assert (m=0xc46595c8, what=0, 
    file=0xc0689a02 "/usr/src/sys/vm/vm_page.c", line=762)
    at /usr/src/sys/kern/kern_mutex.c:855
#12 0xc05f9f7c in vm_page_alloc (object=0xc46595c8, pindex=0, req=0)
    at /usr/src/sys/vm/vm_page.c:762
#13 0xc05ecd4b in vm_fault_copy_entry (dst_map=0xc1922dc8, src_map=0xc45d4bd0, 
    dst_entry=0xc46a5ac8, src_entry=0x0) at /usr/src/sys/vm/vm_fault.c:1167
#14 0xc05f3221 in vm_map_copy_entry (src_map=0xc45d4bd0, dst_map=0xc1922dc8, 
    src_entry=0xc4695ca8, dst_entry=0xc46a5ac8)
    at /usr/src/sys/vm/vm_map.c:2373
#15 0xc05f3547 in vmspace_fork (vm1=0xc45d4bd0)
    at /usr/src/sys/vm/vm_map.c:2488
#16 0xc05ee30e in vm_forkproc (td=0xc467f390, p2=0xc467ed3c, td2=0xc467f130, 
    flags=20) at /usr/src/sys/vm/vm_glue.c:624
#17 0xc04f7c25 in fork1 (td=0xc467f390, flags=20, pages=0, procp=0xddb95cdc)
    at /usr/src/sys/kern/kern_fork.c:654
#18 0xc04f6c6b in fork (td=0xc467f390, uap=0xddb95d14)
    at /usr/src/sys/kern/kern_fork.c:102
#19 0xc063bec3 in syscall (frame=
      {tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 4096, tf_esi = 65536, tf_ebp = -1077946760, tf_isp = -575054476, tf_ebx = 64, tf_edx = 1307, tf_ecx = 672785664, tf_eax = 2, tf_trapno = 0, tf_err = 2, tf_eip = 672191759, tf_cs = 31, tf_eflags = 582, tf_esp = -1077946804, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1006
#20 0xc0624a9d in Xint0x80_syscall () at {standard input}:145
---Can't read userspace from dump, or kernel process---


With a kernel built with sources as of directly after the mentioned commit
the backtrace looks like this:

GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic: vm_fault_copy_wired: page missing
panic messages:
---
panic: vm_fault_copy_wired: page missing
cpuid = 1; lapic.id = 00000000
panic: from debugger
cpuid = 1; lapic.id = 00000000
boot() called on cpu#1
Uptime: 54s
Dumping 512 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496
---
Reading symbols from /boot/kernel/linux.ko...done.
Loaded symbols for /boot/kernel/linux.ko
#0  doadump () at ../../../kern/kern_shutdown.c:240
240             dumping++;
(kgdb) where
#0  doadump () at ../../../kern/kern_shutdown.c:240
#1  0xc0209223 in boot (howto=260) at ../../../kern/kern_shutdown.c:372
#2  0xc0209608 in panic () at ../../../kern/kern_shutdown.c:550
#3  0xc013ed72 in db_panic () at ../../../ddb/db_command.c:450
#4  0xc013ecea in db_command (last_cmdp=0xc03d0c00, cmd_table=0x0, 
    aux_cmd_tablep=0xc038baec, aux_cmd_tablep_end=0xc038baf0)
    at ../../../ddb/db_command.c:346
#5  0xc013edf8 in db_command_loop () at ../../../ddb/db_command.c:472
#6  0xc0141b39 in db_trap (type=3, code=0) at ../../../ddb/db_trap.c:73
#7  0xc031cf41 in kdb_trap (type=3, code=0, regs=0xdc446b44)
    at ../../../i386/i386/db_interface.c:172
#8  0xc033568d in trap (frame=
      {tf_fs = 24, tf_es = 16, tf_ds = 16, tf_edi = -1070069368, tf_esi = 1, tf_ebp = -599495792, tf_isp = -599495824, tf_ebx = 0, tf_edx = 0, tf_ecx = 0, tf_eax = 18, tf_trapno = 3, tf_err = 0, tf_eip = -1070476754, tf_cs = 8, tf_eflags = 642, tf_esp = -1070052523, tf_ss = -1070129405})
    at ../../../i386/i386/trap.c:577
#9  0xc031e848 in calltrap () at {standard input}:103
#10 0xc02095a1 in panic (fmt=0xc0380988 "vm_fault_copy_wired: page missing")
    at ../../../kern/kern_shutdown.c:534
#11 0xc02e7256 in vm_fault_copy_entry (dst_map=0xc1529b00, src_map=0xc437a500, 
    dst_entry=0xc436c924, src_entry=0x0) at ../../../vm/vm_fault.c:1088
#12 0xc02ed6e1 in vm_map_copy_entry (src_map=0xc437a500, dst_map=0xc1529b00,
    src_entry=0xc082ce4c, dst_entry=0xc436c924) at ../../../vm/vm_map.c:2376
#13 0xc02eda07 in vmspace_fork (vm1=0xc437a500) at ../../../vm/vm_map.c:2491
#14 0xc02e878e in vm_forkproc (td=0xc41ce850, p2=0xc41cb000, td2=0xc402e260, 
    flags=20) at ../../../vm/vm_glue.c:624
#15 0xc01f4a78 in fork1 (td=0xc41ce850, flags=20, pages=0, procp=0xdc446cdc)
    at ../../../kern/kern_fork.c:654
#16 0xc01f3a2b in fork (td=0xc41ce850, uap=0xdc446d14)
    at ../../../kern/kern_fork.c:102
#17 0xc0335f83 in syscall (frame=
      {tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134831178, tf_esi = 4096, tf_ebp = -1077946712, tf_isp = -599495308, tf_ebx = 65536, tf_edx = 64, tf_ecx = 672822528, tf_eax = 2, tf_trapno = 0, tf_err = 2, tf_eip = 672228559, tf_cs = 31, tf_eflags = 582, tf_esp = -1077946740, tf_ss = 47})
    at ../../../i386/i386/trap.c:1005
#18 0xc031e89d in Xint0x80_syscall () at {standard input}:145

The change in the backtraces seems to be caused by Marcel Moolenaar's
addition of upward growable stacks on Aug 30 21:25:22 2003 UTC. However,
this is only an assumption based on the differences in sys/vm between
two kernels I tested and not exactly tracked down.

Cdrecord has a single call of mlockall(2) in cdrecord.c:
    364 #if defined(HAVE_MLOCKALL)
    365                 if (mlockall(MCL_CURRENT|MCL_FUTURE) < 0) {
    366                         errmsg("WARNING: Cannot do mlockall(2).\n");
    367                         errmsgno(EX_BAD, "WARNING: This causes a high risk for buffer underruns.\n");
    368                 }
    369 #endif

This looks like a fairly normal use of mlockall(2) in cdrecord and is
known to work on e.g. Solaris and Linux.

Notes:	- I only have tested this on SMP-boxes.
	- There are several other reports of this panic, however, it was
	  first thought to be caused by the ATAng commit. See the archive
	  of freebsd-current.
	- A workaround (patch-conf::configure) was commited to sysutils/
	  cdrtools-devel port that hinders cdrecord from using mlockall(2).
>How-To-Repeat:
Build the sysutils/cdrtools port on a FreeBSD built after the addition of
mlockall(2) and try to write a CD:
	cdrecord dev=<your burner> <some ISO-image>
>Fix:
Reverting the addition of mlockall(2) and munlockall(2) or hacking cdrecord
to not use mlockall(2) avoids the panic.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->bms 
Responsible-Changed-By: bms 
Responsible-Changed-When: Sun 5 Oct 2003 15:34:06 PDT 
Responsible-Changed-Why:  
I'll attempt to reproduce this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=57611 
State-Changed-From-To: open->feedback 
State-Changed-By: bms 
State-Changed-When: Wed 8 Oct 2003 00:46:59 PDT 
State-Changed-Why:  
It's possible that revision 1.180 of src/sys/vm/vm_fault.c fixes this. 
Can you reproduce the problem with this revision? Regrettably I can't 
reproduce the problem as I don't have access to a dev machine with a CD 
burner just now. Thanks. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=57611 

From: Marius Strobl <marius@alchemy.franken.de>
To: Bruce M Simpson <bms@FreeBSD.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/57611: mlockall(2) related VM-bug
Date: Wed, 8 Oct 2003 21:21:10 +0200

 On Wed, Oct 08, 2003 at 12:48:12AM -0700, Bruce M Simpson wrote:
 > 
 > It's possible that revision 1.180 of src/sys/vm/vm_fault.c fixes this.
 > Can you reproduce the problem with this revision? Regrettably I can't
 > reproduce the problem as I don't have access to a dev machine with a CD
 > burner just now. Thanks.
 > 
 
 It hasn't. But with a kernel built from up-to-date sources the backtrace
 has changed again (back to a 'panic: vm_fault_copy_wired: page missing'):
 
 GNU gdb 5.2.1 (FreeBSD)
 Copyright 2002 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-undermydesk-freebsd"...
 panic: vm_fault_copy_wired: page missing
 panic messages:
 ---
 panic: vm_fault_copy_wired: page missing
 cpuid = 0; lapic.id = 01000000
 panic: from debugger
 cpuid = 0; lapic.id = 01000000
 boot() called on cpu#0
 Uptime: 4m41s
 Dumping 511 MB
  16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496
 ---
 Reading symbols from /boot/kernel/linux.ko...done.
 Loaded symbols for /boot/kernel/linux.ko
 #0  doadump () at ../../../kern/kern_shutdown.c:240
 240             dumping++;
 (kgdb) where
 #0  doadump () at ../../../kern/kern_shutdown.c:240
 #1  0xc0533bcc in boot (howto=260) at ../../../kern/kern_shutdown.c:372
 #2  0xc0533ffa in panic () at ../../../kern/kern_shutdown.c:550
 #3  0xc044b832 in db_panic () at ../../../ddb/db_command.c:450
 #4  0xc044b7aa in db_command (last_cmdp=0xc073d400, cmd_table=0x0, 
     aux_cmd_tablep=0xc06f4d38, aux_cmd_tablep_end=0xc06f4d3c)
     at ../../../ddb/db_command.c:346
 #5  0xc044b8b8 in db_command_loop () at ../../../ddb/db_command.c:472
 #6  0xc044e639 in db_trap (type=3, code=0) at ../../../ddb/db_trap.c:73
 #7  0xc0688783 in kdb_trap (type=3, code=0, regs=0xddc2db44)
     at ../../../i386/i386/db_interface.c:171
 #8  0xc06a1336 in trap (frame=
       {tf_fs = 24, tf_es = 16, tf_ds = 16, tf_edi = -1066485554, tf_esi = 1, tf_ebp = -574432368, tf_isp = -574432400, tf_ebx = 0, tf_edx = 0, tf_ecx = 0, tf_eax = 18, tf_trapno = 3, tf_err = 0, tf_eip = -1066890642, tf_cs = 8, tf_eflags = 642, tf_esp = -1066472185, tf_ss = -1066544104})
     at ../../../i386/i386/trap.c:578
 #9  0xc068a088 in calltrap () at {standard input}:103
 #10 0xc0533f51 in panic (fmt=0xc06eb8ce "vm_fault_copy_wired: page missing")
     at ../../../kern/kern_shutdown.c:534
 #11 0xc064f949 in vm_fault_copy_entry (dst_map=0xc4bdd1f8, src_map=0xc4bdd2f4, 
     dst_entry=0xc4c03f00, src_entry=0x0) at ../../../vm/vm_fault.c:1187
 #12 0xc06568c1 in vm_map_copy_entry (src_map=0xc4bdd2f4, dst_map=0xc4bdd1f8,
     src_entry=0xc101dec4, dst_entry=0xc4c03f00) at ../../../vm/vm_map.c:2379
 #13 0xc0656bfb in vmspace_fork (vm1=0xc4bdd2f4) at ../../../vm/vm_map.c:2494
 #14 0xc06513fe in vm_forkproc (td=0xc4968720, p2=0xc4ab6974, td2=0xc4ab7720, 
     flags=20) at ../../../vm/vm_glue.c:624
 #15 0xc051cff0 in fork1 (td=0xc4968720, flags=20, pages=0, procp=0xddc2dcdc)
     at ../../../kern/kern_fork.c:654
 #16 0xc051baab in fork (td=0xc4968720, uap=0xddc2dd14)
     at ../../../kern/kern_fork.c:102
 #17 0xc06a1cb0 in syscall (frame=
       {tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 4096, tf_esi = 65536, tf_ebp = -1077946792, tf_isp = -574431884, tf_ebx = 64, tf_edx = 1307, tf_ecx = 672785664, tf_eax = 2, tf_trapno = 0, tf_err = 2, tf_eip = 672191759, tf_cs = 31, tf_eflags = 582, tf_esp = -1077946836, tf_ss = 47})
     at ../../../i386/i386/trap.c:1006
 #18 0xc068a0dd in Xint0x80_syscall () at {standard input}:145
 ---Can't read userspace from dump, or kernel process---
 
State-Changed-From-To: feedback->patched 
State-Changed-By: bms 
State-Changed-When: Wed 15 Oct 2003 02:27:14 PDT 
State-Changed-Why:  
We believe that revision 1.181 of vm_fault.c fixes the problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=57611 

From: Marius Strobl <marius@alchemy.franken.de>
To: Bruce M Simpson <bms@FreeBSD.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/57611: mlockall(2) related VM-bug
Date: Wed, 15 Oct 2003 23:03:37 +0200

 On Wed, Oct 15, 2003 at 02:28:31AM -0700, Bruce M Simpson wrote:
 > 
 > We believe that revision 1.181 of vm_fault.c fixes the problem.
 > 
 
 I can confirm this.
 
State-Changed-From-To: patched->closed 
State-Changed-By: bms 
State-Changed-When: Thu 16 Oct 2003 12:26:40 PDT 
State-Changed-Why:  
Originator confirms that the issue is fixed. 
Will keep an eye on this in case of regression. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=57611 
>Unformatted:
