From nobody@FreeBSD.org  Wed Jun 17 13:27:29 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B8CEE106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 17 Jun 2009 13:27:29 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id A5F9A8FC1C
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 17 Jun 2009 13:27:29 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n5HDRTqT032915
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 17 Jun 2009 13:27:29 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n5HDRTVY032881;
	Wed, 17 Jun 2009 13:27:29 GMT
	(envelope-from nobody)
Message-Id: <200906171327.n5HDRTVY032881@www.freebsd.org>
Date: Wed, 17 Jun 2009 13:27:29 GMT
From: Alex Urbanowicz <alex.urbanowicz@artegence.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: FreeBSD 8.0-CURRENT #0 r: Tue Jun 16 13:53:39 UTC 2009     alex@dev32:/usr/obj/usr/home/alex/bsd/sys/XEN
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         135667
>Category:       kern
>Synopsis:       ufs filesystem corruption on XEN DomU system
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-xen
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jun 17 13:30:05 UTC 2009
>Closed-Date:    
>Last-Modified:  Thu Dec 30 01:50:09 UTC 2010
>Originator:     Alex Urbanowicz
>Release:        8.0-CURRENT
>Organization:
Artegence
>Environment:
FreeBSD 8.0-CURRENT #0 r: Tue Jun 16 13:53:39 UTC 2009 alex@dev32:/usr/obj/usr/home/alex/bsd/sys/XEN i386

>Description:
Disk operations lead to syscall errors and filesystem corruption. Below are syscall errors generated by system boot.

lock order reversal:
 1st 0xc3a8b8b8 ufs (ufs) @ /usr/home/alex/bsd/sys/kern/vfs_mount.c:1054
 2nd 0xc3a8d6a0 devfs (devfs) @ /usr/home/alex/bsd/sys/kern/vfs_subr.c:2084
KDB: stack backtrace:
X_db_sym_numargs(c0356957,c35b8824,c0116b95,c01072bb,c035979e,...) at X_db_sym_numargs+0x146
kdb_backtrace(c01072bb,c035979e,c36ccbd0,c36ccb00,c35b8880,...) at kdb_backtrace+0x29
witness_display_spinlock(c035979e,c3a8d6a0,c03489cb,c36ccb00,c0360aae,...) at witness_display_spinlock+0x75
witness_checkorder(c3a8d6a0,9,c0360aae,824,0,...) at witness_checkorder+0x839
__lockmgr_args(c3a8d6a0,80100,c3a8d6bc,0,0,...) at __lockmgr_args+0x797
vop_stdlock(c35b8988,c011693b,c0348c10,80100,c3a8d648,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c038f2e0,c35b8988,c39c2524,c03c21a0,c3a8d648,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c3a8d648,80100,c0360aae,824,8,...) at _vn_lock+0x5e
vget(c3a8d648,80100,c39c2480,15d,c0348b28,...) at vget+0xb9
devfs_allocv(c3952000,c394ca10,c35b8a20,9d,c0534698,...) at devfs_allocv+0x102
devfs_rules_apply(c394ca10,80000,c35b8c3c,42c,0,...) at devfs_rules_apply+0x14a
vfs_donmount(c39c2480,0,c3952100,c3952100,bf7fde39,...) at vfs_donmount+0x14bb
nmount(c39c2480,c35b8d08,c,c,c0394bb8,...) at nmount+0x72
syscall(c35b8d48) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x22
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280e7c3b, esp = 0xbf7fde0c, ebp = 0xbf7fe368 ---
lock order reversal:
 1st 0xd772de30 bufwait (bufwait) @ /usr/home/alex/bsd/sys/kern/vfs_bio.c:2558
 2nd 0xc3991e00 dirhash (dirhash) @ /usr/home/alex/bsd/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
X_db_sym_numargs(c0356957,e4d2fa8c,c0116b95,c01072bb,c035979e,...) at X_db_sym_numargs+0x146
kdb_backtrace(c01072bb,c035979e,c36ca4d0,c36ccc38,e4d2fae8,...) at kdb_backtrace+0x29
witness_display_spinlock(c035979e,c3991e00,c0373b42,c36ccc38,c03737d1,...) at witness_display_spinlock+0x75
witness_checkorder(c3991e00,9,c03737d1,11d,0,...) at witness_checkorder+0x839
_sx_xlock(c3991e00,0,c03737d1,11d,d86a3018,...) at _sx_xlock+0x85
ufsdirhash_enduseful(0,d,c38c4800,d772ddd0,d86a3018,...) at ufsdirhash_enduseful+0x2f5
ufsdirhash_remove(c3b2a0e8,d86a3018,18,e4d2fb78,e4d2fb74,...) at ufsdirhash_remove+0x14
ufs_dirremove(c3b25b84,c3b291d0,500800c,1,9f,...) at ufs_dirremove+0xe5
ufs_itimes(e4d2fc44,e4d2fc54,c3b20860,c3b20860,0,...) at ufs_itimes+0x826
VOP_RMDIR_APV(c03b0200,e4d2fc44,2,0,28216238,...) at VOP_RMDIR_APV+0xa5
kern_rmdirat(c3b16000,ffffff9c,28216238,0,e4d2fc90,...) at kern_rmdirat+0x16b
kern_rmdir(c3b16000,28216238,0,e4d2fd3c,c0329e23,...) at kern_rmdir+0x27
rmdir(c3b16000,e4d2fd08,4,c036e7d5,c039315c,...) at rmdir+0x22
syscall(e4d2fd48) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x22
--- syscall (137, FreeBSD ELF32, rmdir), eip = 0x280d8bfb, esp = 0xbf7febfc, ebp = 0xbf7fec28 ---

>How-To-Repeat:
Create a FreeBSD Xen guest on 64-bit CentOS 5.3 host.
>Fix:


>Release-Note:
>Audit-Trail:

From: Uncle Richy <gould300@googlemail.com>
To: bug-followup@FreeBSD.org, alex.urbanowicz@artegence.com
Cc:  
Subject: Re: kern/135667: [lor] LORs causing ufs filesystem corruption on XEN 
	DomU system
Date: Mon, 14 Dec 2009 19:05:33 +0000

 I can confirm that the problem occurs using 8.0-RELEASE on a CentOS
 5.4 x64 host.
 
 I successfully built my domU using the steps on this site ->
 http://www.ita.com.ua/eng/articles.htm?id=34 and the default XEN
 kernel configuration with no changes.
 
 Here is my domU configuration:
 
 kernel = "/var/lib/xen/images/freebsd8dev.kernel"
 memory = 256
 name = "freebsd8devp"
 vif = ['']
 disk = ['file:/var/lib/xen/images/freebsd8dev.img,hda1,w']
 extra = "boot_verbose=1"
 extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"
 extra += ",kern.hz=100"
 pae=1
 vnc=1
 
 I reproduced the problem by running a kernel rebuild inside the domU
 until this point during make:
 
 In file included from isa_if.c:18:
 ../../../isa/isavar.h:51:11: error: ISO C99 requires whitespace after
 the macro name
 ../../../isa/isavar.h:67:10: error: ISO C99 requires whitespace after
 the macro name
 ../../../isa/isavar.h:69:18: error: ISO C99 requires whitespace after
 the macro name
 *** Error code 1
 
 Looking at isavar.h showed:
 
 51 #define IS<B8>dORDER_PNP
 67  #define I<E9>A_PNP_NDRQ 2
 69  #define ISADMA_RE<D7>D  0x00100000
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: brucec 
Responsible-Changed-When: Fri Sep 24 20:58:27 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135667 
State-Changed-From-To: open->feedback 
State-Changed-By: cperciva 
State-Changed-When: Thu Dec 30 01:40:36 UTC 2010 
State-Changed-Why:  
Move into state feedback pending confirmation of whether this can still be 
reproduced (I think I might have fixed it with one of my recent commits). 


Responsible-Changed-From-To: freebsd-fs->freebsd-xen 
Responsible-Changed-By: cperciva 
Responsible-Changed-When: Thu Dec 30 01:40:36 UTC 2010 
Responsible-Changed-Why:  
Reassign Xen bug to freebsd-xen.  It's very unlikely that this is filesystem 
related. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135667 

From: Colin Percival <cperciva@freebsd.org>
To: bug-followup@FreeBSD.org, alex.urbanowicz@artegence.com
Cc:  
Subject: Re: kern/135667: [lor] LORs causing ufs filesystem corruption on
 XEN DomU system
Date: Wed, 29 Dec 2010 17:40:19 -0800

 Can you reproduce this on a recent tree?  We've fixed lots of Xen bugs recently,
 including some which could cause this sort of corruption.
 
 (The LORs are a red herring, btw.)
 
 -- 
 Colin Percival
 Security Officer, FreeBSD | freebsd.org | The power to serve
 Founder / author, Tarsnap | tarsnap.com | Online backups for the truly paranoid
>Unformatted:
