From dm@home3.dinoex.sub.de  Tue Feb 17 19:30:17 2009
Return-Path: <dm@home3.dinoex.sub.de>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 54D431065676
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 17 Feb 2009 19:30:17 +0000 (UTC)
	(envelope-from dm@home3.dinoex.sub.de)
Received: from uucp.dinoex.sub.de (uucp.dinoex.sub.de [194.45.71.2])
	by mx1.freebsd.org (Postfix) with ESMTP id CC87B8FC18
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 17 Feb 2009 19:30:16 +0000 (UTC)
	(envelope-from dm@home3.dinoex.sub.de)
Received: from home3.dinoex.sub.de (home3.dinoex.sub.de [194.45.71.20])
	by uucp.dinoex.sub.de (8.14.2/8.14.2) with ESMTP id n1HIxK2U069180
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 17 Feb 2009 19:59:35 +0100 (CET)
	(envelope-from dm@home3.dinoex.sub.de)
Received: (from dm@localhost)
	by home3.dinoex.sub.de (8.14.3/8.14.3/Submit) id n1HIxMLU008860;
	Tue, 17 Feb 2009 19:59:22 +0100 (CET)
	(envelope-from dm)
Message-Id: <200902171859.n1HIxMLU008860@home3.dinoex.sub.de>
Date: Tue, 17 Feb 2009 19:59:22 +0100 (CET)
From: Dirk Meyer <dirk.meyer@dinoex.sub.de>
Reply-To: Dirk Meyer <dirk.meyer@dinoex.sub.de>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: triple LOR after hard reboot
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         131784
>Category:       kern
>Synopsis:       [lor] triple LOR after hard reboot
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Feb 17 19:40:00 UTC 2009
>Closed-Date:    
>Last-Modified:  Sat Dec 26 22:40:00 UTC 2009
>Originator:     Dirk Meyer
>Release:        FreeBSD 8.0-STABLE i386
>Organization:
>Environment:
System: FreeBSD book64.dinoex.sub.de 8.0-CURRENT FreeBSD 8.0-CURRENT #5: Tue Feb 17 06:39:57 CET 2009     root@book64.dinoex.sub.de:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:

 lock order reversal:
 1st 0xffffff0003a2d7f8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:424
 2nd 0xfffffffe53e58bc8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2443
 3rd 0xffffff00034fc7f8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:545
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 _witness_debugger() at _witness_debugger+0x2e
 witness_checkorder() at witness_checkorder+0x81e
 __lockmgr_args() at __lockmgr_args+0xc2a
 ffs_lock() at ffs_lock+0x8c
 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
 _vn_lock() at _vn_lock+0x47
 ffs_snapshot() at ffs_snapshot+0x1c09
 ffs_mount() at ffs_mount+0x666
 vfs_donmount() at vfs_donmount+0xe31
 nmount() at nmount+0xa6
 syscall() at syscall+0x1bf
 Xfast_syscall() at Xfast_syscall+0xab

 --- syscall (378, FreeBSD ELF64, nmount), rip = 0x8007a6cec, rsp = 0x7fffffffea48, rbp = 0x800a02530 ---
 lock order reversal:
 1st 0xfffffffe53e58bc8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2443
 2nd 0xffffff0003892330 snaplk (snaplk) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:794
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 _witness_debugger() at _witness_debugger+0x2e
 witness_checkorder() at witness_checkorder+0x81e
 __lockmgr_args() at __lockmgr_args+0xc2a
 ffs_lock() at ffs_lock+0x8c
 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
 _vn_lock() at _vn_lock+0x47
 ffs_snapshot() at ffs_snapshot+0x1ad6
 ffs_mount() at ffs_mount+0x666
 vfs_donmount() at vfs_donmount+0xe31
 nmount() at nmount+0xa6
 syscall() at syscall+0x1bf
 Xfast_syscall() at Xfast_syscall+0xab
 --- syscall (378, FreeBSD ELF64, nmount), rip = 0x8007a6cec, rsp = 0x7fffffffea48, rbp = 0x800a02530 ---

 lock order reversal:
 1st 0xffffff0003892330 snaplk (snaplk) @ /usr/src/sys/kern/vfs_vnops.c:293
 2nd 0xffffff0003a2d7f8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:1588
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 _witness_debugger() at _witness_debugger+0x2e
 witness_checkorder() at witness_checkorder+0x81e
 __lockmgr_args() at __lockmgr_args+0xc2a
 ffs_snapremove() at ffs_snapremove+0xe7
 softdep_releasefile() at softdep_releasefile+0x139
 ufs_inactive() at ufs_inactive+0x1a7
 vinactive() at vinactive+0x72
 vput() at vput+0x22c
 vn_close() at vn_close+0xb9
 vn_closefile() at vn_closefile+0x7d
 _fdrop() at _fdrop+0x23
 closef() at closef+0x4d
 kern_close() at kern_close+0x10d
 syscall() at syscall+0x1bf
 Xfast_syscall() at Xfast_syscall+0xab
 --- syscall (6, FreeBSD ELF64, close), rip = 0x80083cf8c, rsp = 0x7fffffffea48, rbp = 0 ---

>How-To-Repeat:

	kernel crashes while loading snd_hda, no crashdump can be written.
 	on reboot ffsck runs (and backroung fsck) which tiggered this LOR.

>Fix:

	unknown

>Release-Note:
>Audit-Trail:

From: Daan Vreeken <Daan@vehosting.nl>
To: bug-followup@freebsd.org
Cc: Dirk Meyer <dm@home3.dinoex.sub.de>
Subject: Re: kern/131784: triple LOR after hard reboot
Date: Sat, 21 Feb 2009 01:27:14 +0100

 On Tuesday 17 February 2009 19:59:22 Dirk Meyer wrote:
 > >Number:         131784
 > >Category:       kern
 > >Synopsis:       triple LOR after hard reboot
 ...
 >  lock order reversal:
 >  1st 0xffffff0003a2d7f8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:424
 >  2nd 0xfffffffe53e58bc8 bufwait (bufwait) @
 ...
 > >How-To-Repeat:
 >
 > 	kernel crashes while loading snd_hda, no crashdump can be written.
 >  	on reboot ffsck runs (and backroung fsck) which tiggered this LOR.
 
 I'm seeing a similar (but slightly different) LOR when booting after loosing 
 the last bit of power of a laptop battery :
 
 lock order reversal:
  1st 0xc5f70ad0 ufs (ufs) @ /mnt/1/src-current/sys/ufs/ffs/ffs_snapshot.c:424
  2nd 0xd9637810 bufwait (bufwait) @ /mnt/1/src-current/sys/kern/vfs_bio.c:1783
  3rd 0xc5b04058 ufs (ufs) @ /mnt/1/src-current/sys/ufs/ffs/ffs_snapshot.c:545
 KDB: stack backtrace:
 db_trace_self_wrapper(c0be832c,e7dbd40c,c0879735,4,c0be37fb,...) at 
 db_trace_self_wrapper+0x26
 kdb_backtrace(4,c0be37fb,c5523740,c55264c0,e7dbd468,...) at kdb_backtrace+0x29
 _witness_debugger(c0beb0e2,c5b04058,c0bde712,c55264c0,c0c082e2,...) at 
 _witness_debugger+0x25
 witness_checkorder(c5b04058,9,c0c082e2,221,0,...) at witness_checkorder+0x839
 __lockmgr_args(c5b04058,80100,c5b04074,0,0,...) at __lockmgr_args+0x797
 ffs_lock(e7dbd578,c0e7b5a0,c5b010a4,80100,c5b04000,...) at ffs_lock+0x8a
 VOP_LOCK1_APV(c0cee5a0,e7dbd578,e7dbd598,c0d06ec0,c5b04000,...) at 
 VOP_LOCK1_APV+0xb5
 _vn_lock(c5b04000,80100,c0c082e2,221,c5553200,...) at _vn_lock+0x5e
 ffs_snapshot(c5b05500,c5861980,c0c09c5b,15e,3,...) at ffs_snapshot+0x1527
 ffs_mount(c5b05500,c5b01000,c0bf18e3,3d7,c586cb00,...) at ffs_mount+0x146f
 vfs_donmount(c5b01000,211000,c5d54380,c5d54380,bfbfed04,...) at 
 vfs_donmount+0x130e
 nmount(c5b01000,e7dbdcf8,c,e7dbdd38,c0ccbb30,...) at nmount+0xbe
 syscall(e7dbdd38) at syscall+0x2a3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (378, FreeBSD ELF32, nmount), eip = 0x280e6c8b, esp = 0xbfbfeb2c, 
 ebp = 0xbfbfee78 ---
 
 lock order reversal:
  1st 0xd9637810 bufwait (bufwait) @ /mnt/1/src-current/sys/kern/vfs_bio.c:1783
  2nd 0xc5fe9c9c snaplk (snaplk) 
 @ /mnt/1/src-current/sys/ufs/ffs/ffs_snapshot.c:2564
  3rd 0xd9610360 bufwait (bufwait) @ /mnt/1/src-current/sys/kern/vfs_bio.c:2443
 KDB: stack backtrace:
 db_trace_self_wrapper(c0be832c,e7dbcfe0,c0879735,4,c0be37fb,...) at 
 db_trace_self_wrapper+0x26
 kdb_backtrace(4,c0be37fb,c5527360,c5523740,e7dbd03c,...) at kdb_backtrace+0x29
 _witness_debugger(c0beb0e2,d9610360,c0bf0833,c5523740,c0befd8e,...) at 
 _witness_debugger+0x25
 witness_checkorder(d9610360,9,c0befd8e,98b,0,...) at witness_checkorder+0x839
 __lockmgr_args(d9610360,81900,c5f70b20,c0bf0768,50,...) at 
 __lockmgr_args+0x797
 getblk(c5f70a78,c1a2cd,0,4000,0,...) at getblk+0x153
 breadn(c5f70a78,c1a2cd,0,4000,0,...) at breadn+0x44
 bread(c5f70a78,c1a2cd,0,4000,0,...) at bread+0x4c
 ffs_blkfree(c5a1e200,dd793000,c5f70a78,60d2210,0,...) at ffs_blkfree+0x328
 mapacct_ufs2(c5f70a78,c60a1000,c60a5000,dd793000,8e7d80,...) at 
 mapacct_ufs2+0xe2
 indiracct_ufs2(60d1658,0,ff7187f4,ffffffff,8e780c,...) at indiracct_ufs2+0x3f1
 indiracct_ufs2(3a6bd18,0,ff7ff7f3,ffffffff,8e780c,...) at indiracct_ufs2+0x338
 indiracct_ufs2(4d2ace8,0,ffbff7f2,ffffffff,80080c,...) at indiracct_ufs2+0x338
 expunge_ufs2(c0a331c0,2,2,29a,0,...) at expunge_ufs2+0x495
 ffs_snapshot(c5b05500,c5861980,c0c09c5b,15e,3,...) at ffs_snapshot+0x239d
 ffs_mount(c5b05500,c5b01000,c0bf18e3,3d7,c586cb00,...) at ffs_mount+0x146f
 vfs_donmount(c5b01000,211000,c5d54380,c5d54380,bfbfed04,...) at 
 vfs_donmount+0x130e
 nmount(c5b01000,e7dbdcf8,c,e7dbdd38,c0ccbb30,...) at nmount+0xbe
 syscall(e7dbdd38) at syscall+0x2a3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (378, FreeBSD ELF32, nmount), eip = 0x280e6c8b, esp = 0xbfbfeb2c, 
 ebp = 0xbfbfee78 ---
 
 lock order reversal:
  1st 0xc5fe9c9c snaplk (snaplk) @ /mnt/1/src-current/sys/kern/vfs_vnops.c:293
  2nd 0xc5f70ad0 ufs (ufs) @ /mnt/1/src-current/sys/ufs/ffs/ffs_snapshot.c:1588
 KDB: stack backtrace:
 db_trace_self_wrapper(c0be832c,e7dbd8c4,c0879735,4,c0be37fb,...) at 
 db_trace_self_wrapper+0x26
 kdb_backtrace(4,c0be37fb,c5527360,c55264c0,e7dbd920,...) at kdb_backtrace+0x29
 _witness_debugger(c0beb0c9,c5f70ad0,c0bde712,c55264c0,c0c082e2,...) at 
 _witness_debugger+0x25
 witness_checkorder(c5f70ad0,9,c0c082e2,634,0,...) at witness_checkorder+0x839
 __lockmgr_args(c5f70ad0,80000,0,0,0,...) at __lockmgr_args+0x797
 ffs_snapremove(c5f70a78,c5b05500,0,c0bf3105,414,...) at ffs_snapremove+0x11f
 softdep_releasefile(c5b157f8,e7dbdaa8,2,c0e7b570,c0cd03e4,...) at 
 softdep_releasefile+0x3b
 ufs_inactive(e7dbdae8,c5f70aec,c5f70a78,c5f70aec,e7dbdb00,...) at 
 ufs_inactive+0x1bc
 VOP_INACTIVE_APV(c0cee5a0,e7dbdae8,c0bf1f1d,92d,c0d06e80,...) at 
 VOP_INACTIVE_APV+0xa5
 vinactive(c0cee5a0,e7dbdb1c,c0bf1f1d,8b3,129,...) at vinactive+0x8e
 vput(c5f70a78,e7dbdb54,c0bf3105,125,c0d06be0,...) at vput+0x1db
 vn_close(c5f70a78,1,c556c400,c5b01000,e7dbdbe0,...) at vn_close+0xee
 vn_closefile(c5a895e8,c5b01000,3,0,c5a895e8,...) at vn_closefile+0xe9
 _fdrop(c5a895e8,c5b01000,e7dbdc1c,c087957c,0,c5b010a4,c0e7b570,c0cd1bf0,c0be048b,c5d4712c,44f,c0be048b,e7dbdc44,c0841d80,c5d4712c,8,c0be048b,44f) 
 at _fdrop+0x43
 closef(c5a895e8,c5b01000,44f,434,c5a895e8,...) at closef+0x290
 kern_close(c5b01000,4,e7dbdd2c,c0b3d5f3,c5b01000,...) at kern_close+0x11d
 close(c5b01000,e7dbdcf8,4,c0bebe9a,c0cc9850,...) at close+0x1a
 syscall(e7dbdd38) at syscall+0x2a3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (6, FreeBSD ELF32, close), eip = 0x28184573, esp = 0xbfbfeb2c, ebp 
 = 0xbfbfee78 ---
 
 System:
 FreeBSD Racebeest.Vitsch.LAN 8.0-CURRENT FreeBSD 8.0-CURRENT #2: Sat Feb 14 
 01:34:10 CET 2009     
 root@Racebeest.Vitsch.LAN:/usr/obj/mnt/1/src-current/sys/GENERIC  i386
 
 
 Regards,
 -- 
 Daan Vreeken
 VEHosting
 http://VEHosting.nl
 tel: +31-(0)40-7113050 / +31-(0)6-46210825
 KvK nr: 17174380

From: "Bjoern A. Zeeb" <bzeeb+freebsd+lor@zabbadoz.net>
To: Dirk Meyer <dm@home3.dinoex.sub.de>
Cc: bug-followup@FreeBSD.org, Daan Vreeken <Daan@vehosting.nl>
Subject: Re: kern/131784: triple LOR after hard reboot
Date: Mon, 23 Feb 2009 11:46:48 +0000 (UTC)

 On Tue, 17 Feb 2009, Dirk Meyer wrote:
 
 See http://sources.zabbadoz.net/freebsd/lor.html
 
 > lock order reversal:
 > 1st 0xffffff0003a2d7f8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:424
 > 2nd 0xfffffffe53e58bc8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2443
 > 3rd 0xffffff00034fc7f8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:545
 
 LOR #285 (new)
 
 > --- syscall (378, FreeBSD ELF64, nmount), rip = 0x8007a6cec, rsp = 0x7fffffffea48, rbp = 0x800a02530 ---
 > lock order reversal:
 > 1st 0xfffffffe53e58bc8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2443
 > 2nd 0xffffff0003892330 snaplk (snaplk) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:794
 
 LOR #269
 
 > lock order reversal:
 > 1st 0xffffff0003892330 snaplk (snaplk) @ /usr/src/sys/kern/vfs_vnops.c:293
 > 2nd 0xffffff0003a2d7f8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:1588
 
 LOR #240
 
 
 
 
 > lock order reversal:
 >  1st 0xc5f70ad0 ufs (ufs) @ /mnt/1/src-current/sys/ufs/ffs/ffs_snapshot.c:424
 >  2nd 0xd9637810 bufwait (bufwait) @ /mnt/1/src-current/sys/kern/vfs_bio.c:1783
 >  3rd 0xc5b04058 ufs (ufs) @ /mnt/1/src-current/sys/ufs/ffs/ffs_snapshot.c:545
 
 LOR #285
 
 > lock order reversal:
 >  1st 0xd9637810 bufwait (bufwait) @ /mnt/1/src-current/sys/kern/vfs_bio.c:1783
 >  2nd 0xc5fe9c9c snaplk (snaplk) @ /mnt/1/src-current/sys/ufs/ffs/ffs_snapshot.c:2564
 >  3rd 0xd9610360 bufwait (bufwait) @ /mnt/1/src-current/sys/kern/vfs_bio.c:2443
 
 LOR #286 (new)
 
 > lock order reversal:
 >  1st 0xc5fe9c9c snaplk (snaplk) @ /mnt/1/src-current/sys/kern/vfs_vnops.c:293
 >  2nd 0xc5f70ad0 ufs (ufs) @ /mnt/1/src-current/sys/ufs/ffs/ffs_snapshot.c:1588
 
 LOR #240
 
 
 
 
 See http://sources.zabbadoz.net/freebsd/lor.html
 
 -- 
 Bjoern A. Zeeb                      The greatest risk is not taking one.

From: Peter Jeremy <peterjeremy@acm.org>
To: bug-followup@FreeBSD.org, dirk.meyer@dinoex.sub.de
Cc:  
Subject: Re: kern/131784: [lor] triple LOR after hard reboot
Date: Sun, 27 Dec 2009 09:30:44 +1100

 --mP3DRpeJDSE+ciuQ
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 I got the same thing just taking a "dump -L" of a filesystem, running:
 FreeBSD server.vk2pj.dyndns.org 8.0-STABLE FreeBSD 8.0-STABLE #11: Mon Nov =
 30 16:36:34 EST 2009     root@server.vk2pj.dyndns.org:/var/obj/usr/src/sys/=
 server  amd64
 
 lock order reversal:
  1st 0xffffff002db3e9d0 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:424
  2nd 0xffffff8080763388 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
  3rd 0xffffff0028f14098 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:545
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper0x2a
 _witness_debugger() at _witness_debugger0x2c
 witness_checkorder() at witness_checkorder0x66f
 __lockmgr_args() at __lockmgr_args0x475
 ffs_lock() at ffs_lock0x8c
 VOP_LOCK1_APV() at VOP_LOCK1_APV0x52
 _vn_lock() at _vn_lock0x47
 ffs_snapshot() at ffs_snapshot0x1af8
 ffs_mount() at ffs_mount0x652
 vfs_donmount() at vfs_donmount0xc80
 nmount() at nmount0x51
 syscall() at syscall0x1ac
 Xfast_syscall() at Xfast_syscall0xe1
 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x80069bb4c, rsp =3D 0x7f=
 ffffffe518, rbp =3D 0x7fffffffef1e ---
 lock order reversal:
  1st 0xffffff8080763388 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
  2nd 0xffffff000be70630 snaplk (snaplk) @ /usr/src/sys/ufs/ffs/ffs_snapshot=
 =2Ec:794
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper0x2a
 _witness_debugger() at _witness_debugger0x2c
 witness_checkorder() at witness_checkorder0x66f
 __lockmgr_args() at __lockmgr_args0x475
 ffs_lock() at ffs_lock0x8c
 VOP_LOCK1_APV() at VOP_LOCK1_APV0x52
 _vn_lock() at _vn_lock0x47
 ffs_snapshot() at ffs_snapshot0x154d
 ffs_mount() at ffs_mount0x652
 vfs_donmount() at vfs_donmount0xc80
 nmount() at nmount0x51
 syscall() at syscall0x1ac
 Xfast_syscall() at Xfast_syscall0xe1
 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x80069bb4c, rsp =3D 0x7f=
 ffffffe518, rbp =3D 0x7fffffffef1e ---
 lock order reversal:
  1st 0xffffff000be70630 snaplk (snaplk) @ /usr/src/sys/kern/vfs_vnops.c:296
  2nd 0xffffff002db3e9d0 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:1588
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper0x2a
 _witness_debugger() at _witness_debugger0x2c
 witness_checkorder() at witness_checkorder0x66f
 __lockmgr_args() at __lockmgr_args0x475
 ffs_snapremove() at ffs_snapremove0xc6
 softdep_releasefile() at softdep_releasefile0x139
 ufs_inactive() at ufs_inactive0x1da
 VOP_INACTIVE_APV() at VOP_INACTIVE_APV0x4a
 vinactive() at vinactive0x68
 vput() at vput0x1ae
 vn_close() at vn_close0xa1
 vn_closefile() at vn_closefile0x5a
 _fdrop() at _fdrop0x23
 closef() at closef0x4a
 fdfree() at fdfree0x204
 exit1() at exit10x24f
 sys_exit() at sys_exit0xe
 syscall() at syscall0x1ac
 Xfast_syscall() at Xfast_syscall0xe1
 --- syscall (1, FreeBSD ELF64, sys_exit), rip =3D 0x8006eea2c, rsp =3D 0x7f=
 ffffffe378, rbp =3D 0x404ec0 ---
 
 If anyone wants to look further, the line number references are
 relative to:
 
 __FBSDID("$FreeBSD: src/sys/kern/vfs_bio.c,v 1.568.2.1 2009/08/03 08:13:06 =
 kensmith Exp $");
 SVN rev 196045 on 2009-08-03 08:13:06Z by kensmith
 
 __FBSDID("$FreeBSD: src/sys/kern/vfs_vnops.c,v 1.286.2.2 2009/09/08 14:43:4=
 2 kib Exp $");
 SVN rev 196974 on 2009-09-08 14:43:42Z by kib
 
 __FBSDID("$FreeBSD: src/sys/ufs/ffs/ffs_snapshot.c,v 1.150.2.1 2009/08/03 0=
 8:13:06 kensmith Exp $");
 SVN rev 196045 on 2009-08-03 08:13:06Z by kensmith
 
 --=20
 Peter Jeremy
 
 --mP3DRpeJDSE+ciuQ
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.14 (FreeBSD)
 
 iEYEARECAAYFAks2jpQACgkQ/opHv/APuIf4tQCaAo9rj+mQdqVjj4RqskWLOzfA
 5xYAoMMLv0YiuwdSw79tFZANWPbLpaLo
 =4uC4
 -----END PGP SIGNATURE-----
 
 --mP3DRpeJDSE+ciuQ--
>Unformatted:
