From nobody@FreeBSD.org  Thu Jan 17 00:33:43 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 7179A16A418
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 17 Jan 2008 00:33:43 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 62A3813C46E
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 17 Jan 2008 00:33:43 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m0H0WN6n039024
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 17 Jan 2008 00:32:23 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m0H0WNnG039023;
	Thu, 17 Jan 2008 00:32:23 GMT
	(envelope-from nobody)
Message-Id: <200801170032.m0H0WNnG039023@www.freebsd.org>
Date: Thu, 17 Jan 2008 00:32:23 GMT
From: Jacob Yocom-Piatt <jy-p@fixedpointgroup.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: geli + ZFS + samba starting on boot panics 7.0-BETA4
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         119735
>Category:       kern
>Synopsis:       [zfs] geli + ZFS + samba starting on boot panics 7.0-BETA4
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-fs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jan 17 00:40:02 UTC 2008
>Closed-Date:    Tue Nov 23 16:18:26 UTC 2010
>Last-Modified:  Tue Nov 23 16:18:26 UTC 2010
>Originator:     Jacob Yocom-Piatt
>Release:        7.0-BETA4
>Organization:
>Environment:
FreeBSD databank1.mydomain.com 7.0-BETA4 FreeBSD 7.0-BETA4 #0: Sun Dec  2 16:34:41 UTC 2007     root@myers.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
problem occurs on a PowerEdge 1950 with dual Intel(R) Xeon(R) E5310 @
1.60GHz processors and 2 GB RAM, connected via SAS from PERC 5/e PCI-e
card to an MD1000 PowerVault disk enclosure holding 4 x 750 GB disks.

the configuration is that each of the 4 disks is encrypted using geli
with AES-CBC-256 cipher, the 4 mfidX.eli devices are put into a RAIDZ
ZFS pool and then samba serves files from that filesystem. this is,
AFAICT, stable when configured by hand, i.e. after bootup.

however, when attempting to automate some of the setup by putting the
following in /etc/rc.conf the machine will panic and dump memory as soon
as anyone tries to access files via samba:

geli_devices="mfid1 mfid2 mfid3 mfid4"
geli_mfid1_flags="-k /root/p_a.key"
geli_mfid2_flags="-k /root/p_a.key"
geli_mfid3_flags="-k /root/p_a.key"
geli_mfid4_flags="-k /root/p_a.key"

zfs_enable="YES"

nmbd_enable="YES"
smbd_enable="YES"
winbindd_enable="YES"

at first i thought this was due to the order in which these items were
listed in /etc/rc.conf but this has proved to be false, the panics occur
independent of ordering. here is a sample of what is displayed prior to
the panic and memory dump:

GEOM_ELI: Detached mfid1.eli on last close.
GEOM_ELI: Detached mfid2.eli on last close.
GEOM: mfid1: corrupt or invalid GPT detected.
GEOM: mfid1: GPT rejected -- may not be recoverable.
GEOM_ELI: Detached mfid3.eli on last close.
GEOM: mfid2: corrupt or invalid GPT detected.
GEOM: mfid2: GPT rejected -- may not be recoverable.
GEOM: mfid3: corrupt or invalid GPT detected.
GEOM: mfid3: GPT rejected -- may not be recoverable.
Jan 15 10:46:17 databank1 root: ZFS: vdev failure, zpool=a type=vdev.open_failed
Jan 15 10:46:17 databank1 last message repeated 2 times
Jan 15 10:46:17 databank1 root: ZFS: vdev failure, zpool=a type=vdev.no_replicas

databank1# panic: ZFS: I/O failure (write on <unknown> off 0: zio 0xffffff0001d7d560 [L0 DMU dnode] 4000L/1000P DVA[0]=<0:4000454000:2000> DVA[1]=<0:c400344000:2000> fletcher4 lzjb LE contiguous birth=351331 fill=32 cksum=13866514693:2802016f659a3:35148240b57cbfd:4a32062c9
cpuid = 3
Uptime: 14h6m10s
Physical memory: 2035 MB
Dumping 296 MB: 281 265 249 233 217 201 185 169 153 137 121 105 89 73 57 41 25 9
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort

>How-To-Repeat:
make sure to have the following in /etc/rc.conf:

geli_devices="mfid1 mfid2 mfid3 mfid4"
geli_mfid1_flags="-k /root/p_a.key"
geli_mfid2_flags="-k /root/p_a.key"
geli_mfid3_flags="-k /root/p_a.key"
geli_mfid4_flags="-k /root/p_a.key"

zfs_enable="YES"

nmbd_enable="YES"
smbd_enable="YES"
winbindd_enable="YES"

it prompts for the disk passphrase fine, shows ZFS starting up and starts,
albeit with a delay, the samba daemons. once access is attempted via samba
the machine panics and dumps memory.

i do not have DDB via serial support compiled into the kernel so i cannot
provide a bt or ps output at the moment. these outputs and/or memory dumps
can be furnished upon request, best done over a weekend.
>Fix:
do not have any of the above entries in /etc/rc.conf and configure it all
"by hand" after boot:

# geli attach -k /root/p_a.key /dev/mfid1
..
(enter passphrases and repeat for other disks)
..
# zpool status (to ensure the FS is up)
# /usr/local/sbin/smbd
# /usr/local/sbin/nmbd
# /usr/local/sbin/winbindd

and then all runs fine.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->pjd 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Sat Mar 8 20:41:19 UTC 2008 
Responsible-Changed-Why:  
Assign to Pawel, owner of ZFS and GELI. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119735 
State-Changed-From-To: open->feedback 
State-Changed-By: pjd 
State-Changed-When: pon 10 mar 12:33:00 2008 UTC 
State-Changed-Why:  
Your geli providers are configured to detach on last close. 
Don't do that with ZFS, as it may open/close providers many times. 
Try adding geli_autodetach="NO" to your /etc/rc.conf and see if 
that helps. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119735 
Responsible-Changed-From-To: pjd->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Thu May 28 22:22:00 UTC 2009 
Responsible-Changed-Why:  
With pjd's permission, reassing ZFS-related PRs to freebsd-fs. 

To submitter: did the suggestion in the PR solve your problem? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119735 
State-Changed-From-To: feedback->closed 
State-Changed-By: jh 
State-Changed-When: Tue Nov 23 16:18:24 UTC 2010 
State-Changed-Why:  
Feedback timeout. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119735 
>Unformatted:
