From nobody@FreeBSD.org  Sun May  4 13:39:27 2014
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id DD6D08D6
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  4 May 2014 13:39:27 +0000 (UTC)
Received: from cgiserv.freebsd.org (cgiserv.freebsd.org [IPv6:2001:1900:2254:206a::50:4])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id CB542156E
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  4 May 2014 13:39:27 +0000 (UTC)
Received: from cgiserv.freebsd.org ([127.0.1.6])
	by cgiserv.freebsd.org (8.14.8/8.14.8) with ESMTP id s44DdRIK085202
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 4 May 2014 13:39:27 GMT
	(envelope-from nobody@cgiserv.freebsd.org)
Received: (from nobody@localhost)
	by cgiserv.freebsd.org (8.14.8/8.14.8/Submit) id s44DdR6D085195;
	Sun, 4 May 2014 13:39:27 GMT
	(envelope-from nobody)
Message-Id: <201405041339.s44DdR6D085195@cgiserv.freebsd.org>
Date: Sun, 4 May 2014 13:39:27 GMT
From: Radim Kolar <hsn@sendmail.cz>
To: freebsd-gnats-submit@FreeBSD.org
Subject: zfs panic on root mount 10-stable
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         189355
>Category:       kern
>Synopsis:       [zfs] zfs panic on root mount 10-stable
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun May 04 13:40:00 UTC 2014
>Closed-Date:    
>Last-Modified:  Fri May 23 17:30:00 UTC 2014
>Originator:     Radim Kolar
>Release:        10-STABLE
>Organization:
Z
>Environment:
i386
>Description:
I upgraded 10.0-RELEASE to 10.0-STABLE, can not mount root anymore zfs panics on boot. After rebooting with 10.0-RELEASE ISO i can zpool import without errors.

I do not have dump, because panic is before filesystem is mounted.

panic: double fault

stacktrace

vdev_queue_io_to_issue
vdev_queue_io
zio_vdev_io_start
zio_execute
vdev_mirror_io_start
zio_vdev_io_start
zio_execute
zio_gang_tree_assemble
zio_gang_assemble
zio_execute
spa_load_verify_cb
traverse_visitbp
traverse_visitbp
traverse_dnode
traverse_visitbp
traverse_visitbp
>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:

From: Radim Kolar <hsn@sendmail.cz>
To: "bug-followup@FreeBSD.org" <bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: zfs panic on root mount 10-stable
Date: Sun, 4 May 2014 14:38:14 +0000

  	<201405041339.s44DdR6D085195@cgiserv.freebsd.org>,<201405041340.s44De0NF098695@freefall.freebsd.org>
 MIME-Version: 1.0
 
 --_ed045364-21af-4439-bdc5-79a84d91ed12_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 freebsd revision is r265265
 
 --_ed045364-21af-4439-bdc5-79a84d91ed12_--
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sun May 4 18:08:31 UTC 2014 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=189355 

From: "Steven Hartland" <killing@multiplay.co.uk>
To: <bug-followup@freebsd.org>,
	<hsn@sendmail.cz>
Cc:  
Subject: Re: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Sun, 4 May 2014 19:16:43 +0100

 What version of stable, what zpool layout are you using
 on what hardware?
 
     Regards
     Steve

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Sun, 4 May 2014 22:54:34 +0000

 --_7e6d6f21-3c4f-4028-b48b-906f77411ad3_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 10.0-STABLE #0 r265265 running in VMware. i run this virtual machine nice f=
 reebsd 7.
 
 Problems with ZFS started with 10.0 release. Sometimes it panicked at boot =
 during initial mount after unclean shutdown=2C but restarting virtual machi=
 ne fixed problem.
 
 Reported problem is after good shutdown and update to 10-STABLE. ZFS versio=
 n 5=2C pool version 5000. Architecture is i386 (32-bit).
 
 If you know how to make panic dump work during initial mount let me know i =
 will capture more data.
  		 	   		  =
 
 --_7e6d6f21-3c4f-4028-b48b-906f77411ad3_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>10.0-STABLE #0 r265265 running i=
 n VMware. i run this virtual machine nice freebsd 7.<br><br>Problems with Z=
 FS started with 10.0 release. Sometimes it panicked at boot during initial =
 mount after unclean shutdown=2C but restarting virtual machine fixed proble=
 m.<br><br>Reported problem is after good shutdown and update to 10-STABLE. =
 ZFS version 5=2C pool version 5000. Architecture is i386 (32-bit).<br><br>I=
 f you know how to make panic dump work during initial mount let me know i w=
 ill capture more data.<br> 		 	   		  </div></body>
 </html>=
 
 --_7e6d6f21-3c4f-4028-b48b-906f77411ad3_--

From: "Steven Hartland" <killing@multiplay.co.uk>
To: <bug-followup@freebsd.org>,
	<hsn@sendmail.cz>
Cc:  
Subject: Re: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Tue, 6 May 2014 01:44:29 +0100

 This is a multi-part message in MIME format.
 
 ------=_NextPart_000_0300_01CF68CC.B8749960
 Content-Type: text/plain;
 	format=flowed;
 	charset="Windows-1252";
 	reply-type=original
 Content-Transfer-Encoding: 7bit
 
 Can you try building a debug kernel with the attached patch.
 
 It should allow you to configure a dump device before 
 the root device is mounted by specifying it in /boot/loader.conf
 kern.shutdown.defaultdumpdev="ada4p3"
 
 Ensure you choose a valid device such a swap device.
 
 If all goes well that will enable you to get a proper stack
 trace from the dump.
 
     Regards
     Steve
 ------=_NextPart_000_0300_01CF68CC.B8749960
 Content-Type: application/octet-stream;
 	name="default-dump-dev.patch"
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment;
 	filename="default-dump-dev.patch"
 
 Quick patch which configures kernel dump location prior to mounting root.=0A=
 =0A=
 The location is controlled by the new tunable:=0A=
 kern.shutdown.defaultdumpdev=0A=
 =0A=
 An example of configuring it would be to add the following to =
 /boot/loader.conf:=0A=
 kern.shutdown.defaultdumpdev=3D"ada4p3"=0A=
 =0A=
 This would configure kernel dumps on ata disk 4 partition 3.=0A=
 =0A=
 The usual rules should be maintained when picking a device i.e. choose a =
 device=0A=
 use for swap or otherwise unused.=0A=
 --- sys/kern/kern_shutdown.c.orig	2014-05-04 23:37:01.954116628 +0000=0A=
 +++ sys/kern/kern_shutdown.c	2014-05-06 00:28:54.591101862 +0000=0A=
 @@ -50,12 +50,14 @@ __FBSDID("$FreeBSD: releng/10.0/sys/kern=0A=
  #include <sys/conf.h>=0A=
  #include <sys/cons.h>=0A=
  #include <sys/eventhandler.h>=0A=
 +#include <sys/fcntl.h>=0A=
  #include <sys/jail.h>=0A=
  #include <sys/kdb.h>=0A=
  #include <sys/kernel.h>=0A=
  #include <sys/kerneldump.h>=0A=
  #include <sys/kthread.h>=0A=
  #include <sys/ktr.h>=0A=
 +#include <sys/limits.h>=0A=
  #include <sys/malloc.h>=0A=
  #include <sys/mount.h>=0A=
  #include <sys/priv.h>=0A=
 @@ -72,6 +74,9 @@ __FBSDID("$FreeBSD: releng/10.0/sys/kern=0A=
  =0A=
  #include <ddb/ddb.h>=0A=
  =0A=
 +#include <fs/devfs/devfs_int.h>=0A=
 +#include <geom/geom.h>=0A=
 +=0A=
  #include <machine/cpu.h>=0A=
  #include <machine/pcb.h>=0A=
  #include <machine/smp.h>=0A=
 @@ -245,6 +250,72 @@ print_uptime(void)=0A=
  	printf("%lds\n", (long)ts.tv_sec);=0A=
  }=0A=
  =0A=
 +static char defaultdumpdev[MAXPATHLEN];=0A=
 +TUNABLE_STR("kern.shutdown.defaultdumpdev", defaultdumpdev,=0A=
 +    sizeof(defaultdumpdev));=0A=
 +SYSCTL_STRING(_kern_shutdown, OID_AUTO, defaultdumpdev, CTLFLAG_RDTUN,=0A=
 +    defaultdumpdev, 0, "Default device for early kernel dumps");=0A=
 +=0A=
 +int=0A=
 +setdumpdev(char *devname)=0A=
 +{=0A=
 +	struct thread *td =3D curthread;=0A=
 +	int error, i, ref;=0A=
 +	struct g_consumer *cp;=0A=
 +	struct g_kerneldump kd;=0A=
 +	struct cdev_priv *cdp;=0A=
 +	struct cdev *dev;=0A=
 +	struct cdevsw *dsw;=0A=
 +=0A=
 +	if (devname =3D=3D NULL || strlen(devname) =3D=3D 0)=0A=
 +		return (set_dumper(NULL, NULL));=0A=
 +=0A=
 +	dev =3D NULL;=0A=
 +	dev_lock();=0A=
 +	TAILQ_FOREACH(cdp, &cdevp_list, cdp_list) {=0A=
 +		dev =3D &cdp->cdp_c;=0A=
 +		if (strcmp(dev->si_name, devname) =3D=3D 0)=0A=
 +			break;=0A=
 +		dev =3D NULL;=0A=
 +	}=0A=
 +	dev_unlock();=0A=
 +=0A=
 +	if (dev =3D=3D NULL)=0A=
 +		return (ENOENT);=0A=
 +=0A=
 +	dsw =3D dev_refthread(dev, &ref);=0A=
 +	if (dsw =3D=3D NULL)=0A=
 +		return (ENXIO);=0A=
 +=0A=
 +	error =3D dsw->d_open(dev, FREAD, 0, td);=0A=
 +	if (error !=3D 0) {=0A=
 +		dev_relthread(dev, ref);=0A=
 +		return (error);=0A=
 +	}=0A=
 +=0A=
 +	cp =3D dev->si_drv2;=0A=
 +	kd.offset =3D 0;=0A=
 +	kd.length =3D OFF_MAX;=0A=
 +	i =3D sizeof(kd);=0A=
 +	error =3D g_io_getattr("GEOM::kerneldump", cp, &i, &kd);=0A=
 +	if (error =3D=3D 0) {=0A=
 +		error =3D set_dumper(&kd.di, devtoname(dev));=0A=
 +		if (error =3D=3D 0)=0A=
 +			dev->si_flags |=3D SI_DUMPDEV;=0A=
 +	}=0A=
 +=0A=
 +	(void)dev->si_devsw->d_close(dev, FREAD, 0, td);=0A=
 +	dev_relthread(dev, ref);=0A=
 +=0A=
 +	return (error);=0A=
 +}=0A=
 +=0A=
 +int=0A=
 +setdumpdev_default(void)=0A=
 +{=0A=
 +	return (setdumpdev(defaultdumpdev));=0A=
 +}=0A=
 +=0A=
  int=0A=
  doadump(boolean_t textdump)=0A=
  {=0A=
 --- sys/kern/init_main.c.orig	2014-05-05 18:06:24.008837474 +0000=0A=
 +++ sys/kern/init_main.c	2014-05-05 22:39:52.964175470 +0000=0A=
 @@ -697,6 +697,8 @@ start_init(void *dummy)=0A=
  	struct thread *td;=0A=
  	struct proc *p;=0A=
  =0A=
 +	setdumpdev_default();=0A=
 +=0A=
  	mtx_lock(&Giant);=0A=
  =0A=
  	GIANT_REQUIRED;=0A=
 --- sys/sys/conf.h.orig	2014-05-05 02:20:24.408440686 +0000=0A=
 +++ sys/sys/conf.h	2014-05-05 15:21:06.150967151 +0000=0A=
 @@ -338,6 +338,8 @@ int set_dumper(struct dumperinfo *, cons=0A=
  int dump_write(struct dumperinfo *, void *, vm_offset_t, off_t, size_t);=0A=
  void dumpsys(struct dumperinfo *);=0A=
  int doadump(boolean_t);=0A=
 +int setdumpdev_default(void);=0A=
 +int setdumpdev(char *);=0A=
  extern int dumping;		/* system is dumping */=0A=
  =0A=
  #endif /* _KERNEL */=0A=
 
 ------=_NextPart_000_0300_01CF68CC.B8749960--
 

From: "Steven Hartland" <killing@multiplay.co.uk>
To: <bug-followup@freebsd.org>,
	<hsn@sendmail.cz>
Cc:  
Subject: Re: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Wed, 14 May 2014 15:11:10 +0100

 Any luck with the patch Radim?

From: Radim Kolar <hsn@sendmail.cz>
To: "bug-followup@FreeBSD.org" <bug-followup@freebsd.org>,
	"killing@multiplay.co.uk" <killing@multiplay.co.uk>
Cc:  
Subject: Re: kern/189355: zfs panic on 10-stable
Date: Thu, 15 May 2014 12:38:20 +0000

 --_a717f13e-e8ab-4cae-9f7f-5ec14329a6c1_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 With your path=2C i was able to capture crash dump. This patch should be ad=
 ded to mainstream bsd to help people with capture boot panics.
 
 Here is beginning of core.txt file. Let me know if you need more info.
 
 Trying to mount root from zfs:root []...
 
 Fatal double fault:
 eip =3D 0xc0cb801e
 esp =3D 0xd93eff98
 ebp =3D 0xd93f0300
 cpuid =3D 0=3B apic id =3D 00
 panic: double fault
 cpuid =3D 0
 KDB: enter: panic
 
 Reading symbols from /boot/kernel/zfs.ko.symbols...done.
 Loaded symbols for /boot/kernel/zfs.ko.symbols
 Reading symbols from /boot/kernel/krpc.ko.symbols...done.
 Loaded symbols for /boot/kernel/krpc.ko.symbols
 Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
 Loaded symbols for /boot/kernel/opensolaris.ko.symbols
 Reading symbols from /boot/kernel/if_vmx.ko.symbols...done.
 Loaded symbols for /boot/kernel/if_vmx.ko.symbols
 Reading symbols from /boot/kernel/cc_hd.ko.symbols...done.
 Loaded symbols for /boot/kernel/cc_hd.ko.symbols
 Reading symbols from /boot/kernel/h_ertt.ko.symbols...done.
 Loaded symbols for /boot/kernel/h_ertt.ko.symbols
 #0  doadump (textdump=3D0) at pcpu.h:233
 233     pcpu.h: No such file or directory.
         in pcpu.h
 (kgdb) #0  doadump (textdump=3D0) at pcpu.h:233
 #1  0xc04c87d1 in db_dump (dummy=3D-1066847139=2C dummy2=3D0=2C dummy3=3D-1=
 =2C
     dummy4=3D0xc0a808b4 "") at /usr/src/sys/ddb/db_command.c:543
 #2  0xc04c82cb in db_command (cmd_table=3D<value optimized out>)
     at /usr/src/sys/ddb/db_command.c:449
 #3  0xc04c8010 in db_command_loop () at /usr/src/sys/ddb/db_command.c:502
 #4  0xc04ca8a1 in db_trap (type=3D<value optimized out>=2C
     code=3D<value optimized out>) at /usr/src/sys/ddb/db_main.c:231
 #5  0xc0693bd4 in kdb_trap (type=3D<value optimized out>=2C
     code=3D<value optimized out>=2C tf=3D<value optimized out>)
     at /usr/src/sys/kern/subr_kdb.c:656
 #6  0xc093918f in trap (frame=3D<value optimized out>)
     at /usr/src/sys/i386/i386/trap.c:712
 #7  0xc092336c in calltrap () at /usr/src/sys/i386/i386/exception.s:170
 #8  0xc069345d in kdb_enter (why=3D0xc09a7ef8 "panic"=2C
     msg=3D<value optimized out>) at cpufunc.h:71
 #9  0xc065710f in panic (fmt=3D<value optimized out>)
     at /usr/src/sys/kern/kern_shutdown.c:823
 #10 0xc0939acb in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:1072
 #11 0xc0cb801e in vdev_queue_io_to_issue (vq=3D0xc46a4b00)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/vdev_queue.c:471
 #12 0xc0cb7fb8 in vdev_queue_io (zio=3D0xc4855000)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/vdev_queue.c:744
 #13 0xc0cd84ee in zio_vdev_io_start (ziop=3D<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/zio.c:2607
 #14 0xc0cd4c18 in zio_execute (zio=3D0xc4855000)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/zio.c:1350
 #15 0xc0cb74e4 in vdev_mirror_io_start (zio=3D0xc4823894)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/vdev_mirror.c:284
 
  		 	   		  =
 
 --_a717f13e-e8ab-4cae-9f7f-5ec14329a6c1_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>With your path=2C i was able to =
 capture crash dump. This patch should be added to mainstream bsd to help pe=
 ople with capture boot panics.<br><br>Here is beginning of core.txt file. L=
 et me know if you need more info.<br><br>Trying to mount root from zfs:root=
  []...<br><br>Fatal double fault:<br>eip =3D 0xc0cb801e<br>esp =3D 0xd93eff=
 98<br>ebp =3D 0xd93f0300<br>cpuid =3D 0=3B apic id =3D 00<br>panic: double =
 fault<br>cpuid =3D 0<br>KDB: enter: panic<br><br>Reading symbols from /boot=
 /kernel/zfs.ko.symbols...done.<br>Loaded symbols for /boot/kernel/zfs.ko.sy=
 mbols<br>Reading symbols from /boot/kernel/krpc.ko.symbols...done.<br>Loade=
 d symbols for /boot/kernel/krpc.ko.symbols<br>Reading symbols from /boot/ke=
 rnel/opensolaris.ko.symbols...done.<br>Loaded symbols for /boot/kernel/open=
 solaris.ko.symbols<br>Reading symbols from /boot/kernel/if_vmx.ko.symbols..=
 .done.<br>Loaded symbols for /boot/kernel/if_vmx.ko.symbols<br>Reading symb=
 ols from /boot/kernel/cc_hd.ko.symbols...done.<br>Loaded symbols for /boot/=
 kernel/cc_hd.ko.symbols<br>Reading symbols from /boot/kernel/h_ertt.ko.symb=
 ols...done.<br>Loaded symbols for /boot/kernel/h_ertt.ko.symbols<br>#0&nbsp=
 =3B doadump (textdump=3D0) at pcpu.h:233<br>233&nbsp=3B&nbsp=3B&nbsp=3B&nbs=
 p=3B pcpu.h: No such file or directory.<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B=
 &nbsp=3B&nbsp=3B&nbsp=3B in pcpu.h<br>(kgdb) #0&nbsp=3B doadump (textdump=
 =3D0) at pcpu.h:233<br>#1&nbsp=3B 0xc04c87d1 in db_dump (dummy=3D-106684713=
 9=2C dummy2=3D0=2C dummy3=3D-1=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B dummy4=3D0xc0=
 a808b4 "") at /usr/src/sys/ddb/db_command.c:543<br>#2&nbsp=3B 0xc04c82cb in=
  db_command (cmd_table=3D&lt=3Bvalue optimized out&gt=3B)<br>&nbsp=3B&nbsp=
 =3B&nbsp=3B at /usr/src/sys/ddb/db_command.c:449<br>#3&nbsp=3B 0xc04c8010 i=
 n db_command_loop () at /usr/src/sys/ddb/db_command.c:502<br>#4&nbsp=3B 0xc=
 04ca8a1 in db_trap (type=3D&lt=3Bvalue optimized out&gt=3B=2C<br>&nbsp=3B&n=
 bsp=3B&nbsp=3B code=3D&lt=3Bvalue optimized out&gt=3B) at /usr/src/sys/ddb/=
 db_main.c:231<br>#5&nbsp=3B 0xc0693bd4 in kdb_trap (type=3D&lt=3Bvalue opti=
 mized out&gt=3B=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B code=3D&lt=3Bvalue optimized=
  out&gt=3B=2C tf=3D&lt=3Bvalue optimized out&gt=3B)<br>&nbsp=3B&nbsp=3B&nbs=
 p=3B at /usr/src/sys/kern/subr_kdb.c:656<br>#6&nbsp=3B 0xc093918f in trap (=
 frame=3D&lt=3Bvalue optimized out&gt=3B)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /us=
 r/src/sys/i386/i386/trap.c:712<br>#7&nbsp=3B 0xc092336c in calltrap () at /=
 usr/src/sys/i386/i386/exception.s:170<br>#8&nbsp=3B 0xc069345d in kdb_enter=
  (why=3D0xc09a7ef8 "panic"=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B msg=3D&lt=3Bvalue=
  optimized out&gt=3B) at cpufunc.h:71<br>#9&nbsp=3B 0xc065710f in panic (fm=
 t=3D&lt=3Bvalue optimized out&gt=3B)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/sr=
 c/sys/kern/kern_shutdown.c:823<br>#10 0xc0939acb in dblfault_handler () at =
 /usr/src/sys/i386/i386/trap.c:1072<br>#11 0xc0cb801e in vdev_queue_io_to_is=
 sue (vq=3D0xc46a4b00)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/modules/z=
 fs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:471<br>#12=
  0xc0cb7fb8 in vdev_queue_io (zio=3D0xc4855000)<br>&nbsp=3B&nbsp=3B&nbsp=3B=
  at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/z=
 fs/vdev_queue.c:744<br>#13 0xc0cd84ee in zio_vdev_io_start (ziop=3D&lt=3Bva=
 lue optimized out&gt=3B)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/module=
 s/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2607<br>#14 0x=
 c0cd4c18 in zio_execute (zio=3D0xc4855000)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /=
 usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zi=
 o.c:1350<br>#15 0xc0cb74e4 in vdev_mirror_io_start (zio=3D0xc4823894)<br>&n=
 bsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/modules/zfs/../../cddl/contrib/opens=
 olaris/uts/common/fs/zfs/vdev_mirror.c:284<br><br> 		 	   		  </div></body>
 </html>=
 
 --_a717f13e-e8ab-4cae-9f7f-5ec14329a6c1_--

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "freebsd-fs@FreeBSD.org"
	<freebsd-fs@freebsd.org>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: zfs panic on 10-stable
Date: Thu, 15 May 2014 14:50:55 +0000

 --_eb78ed80-5e1c-4bbf-9389-5250a3119c5e_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 > This could be something that has already has a fixed in head but wasn't e=
 xpected
 > to get triggered without TRIM queueing which isn't in 10-stable yet. The =
 fix
 > is also likely to change based on feedback from the openzfs guys=2C hence=
  isn't
 > in 10-stable yet.
 
 only workaround to get 10-STABLE boot without panic is to boot 10.0 and the=
 n import/export pool.
 
 265046 is already merged in 10-stable.
 Now building stable with 265152 and 265321 merged.
  		 	   		  =
 
 --_eb78ed80-5e1c-4bbf-9389-5250a3119c5e_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>&gt=3B This could be something t=
 hat has already has a fixed in head but wasn't expected<br><div>&gt=3B to g=
 et triggered without TRIM queueing which isn't in 10-stable yet. The fix<br=
 >&gt=3B is also likely to change based on feedback from the openzfs guys=2C=
  hence isn't<br>&gt=3B in 10-stable yet.<br><br>only workaround to get 10-S=
 TABLE boot without panic is to boot 10.0 and then import/export pool.<br><b=
 r>265046 is already merged in 10-stable.<br>Now building stable with 265152=
  and 265321 merged.<br></div> 		 	   		  </div></body>
 </html>=
 
 --_eb78ed80-5e1c-4bbf-9389-5250a3119c5e_--

From: "Steven Hartland" <killing@multiplay.co.uk>
To: "Radim Kolar" <hsn@sendmail.cz>,
	"freebsd-fs@FreeBSD.org" <freebsd-fs@freebsd.org>,
	<bug-followup@freebsd.org>
Cc:  
Subject: Re: kern/189355: zfs panic on 10-stable
Date: Thu, 15 May 2014 16:05:39 +0100

 ----- Original Message ----- 
 From: "Radim Kolar" <hsn@sendmail.cz>
 
 
 >> This could be something that has already has a fixed in head but wasn't expected
 >> to get triggered without TRIM queueing which isn't in 10-stable yet. The fix
 >> is also likely to change based on feedback from the openzfs guys, hence isn't
 >> in 10-stable yet.
 >
 > only workaround to get 10-STABLE boot without panic is to boot 10.0 and then import/export pool
 >
 >
 > 265046 is already merged in 10-stable.
 
 Ah yes I did merge that one just in case.
 
 > Now building stable with 265152 and 265321 merged.
 
 You shouldn't need 265152 as thats just for queued TRIM's and may just confuse
 things further.
 
 
 I'm not sure 265321 on its own will make a difference here as thats a fix for 
 stack overflow and as your stack in the trace is only 14 deep shouldn't be your
 case.
 
 Once your done building and if it does still panic can you confirm the code line
 at the panic location with kgdb as well as the details of the zio being processed
 under pretty print.

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "freebsd-fs@FreeBSD.org"
	<freebsd-fs@freebsd.org>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: zfs panic on 10-stable
Date: Thu, 15 May 2014 18:18:59 +0000

 --_6580e2cf-a4f8-46f1-bf32-6f82ec9394a2_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 > 265046 is already merged in 10-stable.
 > Now building stable with 265152 and 265321 merged.
 
 these 2 patches did not remove panic on boot. I am compiling kernel with
 makeoptions DEBUG=3D-g WITH_CTF=3D1
 
 are there any options which i can use for creating more debugable dump? suc=
 h as add -O0 somewhere?
  		 	   		   		 	   		  =
 
 --_6580e2cf-a4f8-46f1-bf32-6f82ec9394a2_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>&gt=3B 265046 is already merged =
 in 10-stable.<br><div><div dir=3D"ltr"><div>&gt=3B Now building stable with=
  265152 and 265321 merged.<br><br>these 2 patches did not remove panic on b=
 oot. I am compiling kernel with<br>makeoptions DEBUG=3D-g WITH_CTF=3D1<br><=
 br>are there any options which i can use for creating more debugable dump? =
 such as add -O0 somewhere?<br></div> 		 	   		  </div></div> 		 	   		  </d=
 iv></body>
 </html>=
 
 --_6580e2cf-a4f8-46f1-bf32-6f82ec9394a2_--

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "freebsd-fs@FreeBSD.org"
	<freebsd-fs@freebsd.org>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: zfs panic on 10-stable
Date: Thu, 15 May 2014 18:44:37 +0000

 --_3fd876d8-95d2-45ec-9c06-a4c1351e561f_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 crash stack from patched kernel
 
 #7  0xc092336c in calltrap () at /usr/src/sys/i386/i386/exception.s:170
 #8  0xc069345d in kdb_enter (why=3D0xc09a7ef8 "panic"=2C
     msg=3D<value optimized out>) at cpufunc.h:71
 #9  0xc065710f in panic (fmt=3D<value optimized out>)
     at /usr/src/sys/kern/kern_shutdown.c:823
 #10 0xc0939acb in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:1072
 #11 0xc0cb8187 in vdev_queue_io_to_issue (vq=3D0xc46ab300)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/vdev_queue.c:489
 #12 0xc0cb8119 in vdev_queue_io (zio=3D0xc486c5b8)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/vdev_queue.c:768
 #13 0xc0cd8704 in zio_vdev_io_start (ziop=3D<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/zio.c:2617
 #14 0xc0cd4e28 in zio_execute (zio=3D0xc486c5b8)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/zio.c:1357
 #15 0xc0cb7604 in vdev_mirror_io_start (zio=3D0xc476f000)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/vdev_mirror.c:284
 #16 0xc0cd854f in zio_vdev_io_start (ziop=3D<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/zio.c:2537
 #17 0xc0cd4e28 in zio_execute (zio=3D0xc476f000)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/zio.c:1357
 #18 0xc0ca0ddc in spa_load_verify_cb (zilog=3D0x0=2C bp=3D<value optimized =
 out>=2C
     dnp=3D0xc4b0fa00=2C arg=3D<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/spa.c:1887
 #19 0xc0c5eb79 in traverse_visitbp (td=3D0xd93f1120=2C dnp=3D0xc46ab300=2C
     bp=3D0xc4b18100=2C zb=3D0xd93f06b0)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/dmu_traverse.c:272
 #20 0xc0c5edb0 in traverse_visitbp (td=3D0xd93f1120=2C dnp=3D0xc4b0fa00=2C =
 bp=3D0x80=2C
     zb=3D0xd93f07a8)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/dmu_traverse.c:306
 
 failing line 489 is:
 
         if (avl_numnodes(&vq->vq_active_tree) >=3D zfs_vdev_max_active)
                 return (ZIO_PRIORITY_NUM_QUEUEABLE)=3B
 
 
  		 	   		  =
 
 --_3fd876d8-95d2-45ec-9c06-a4c1351e561f_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>crash stack from patched kernel<=
 br><br>#7&nbsp=3B 0xc092336c in calltrap () at /usr/src/sys/i386/i386/excep=
 tion.s:170<br>#8&nbsp=3B 0xc069345d in kdb_enter (why=3D0xc09a7ef8 "panic"=
 =2C<br>&nbsp=3B&nbsp=3B&nbsp=3B msg=3D&lt=3Bvalue optimized out&gt=3B) at c=
 pufunc.h:71<br>#9&nbsp=3B 0xc065710f in panic (fmt=3D&lt=3Bvalue optimized =
 out&gt=3B)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/kern/kern_shutdown.c=
 :823<br>#10 0xc0939acb in dblfault_handler () at /usr/src/sys/i386/i386/tra=
 p.c:1072<br>#11 0xc0cb8187 in vdev_queue_io_to_issue (vq=3D0xc46ab300)<br>&=
 nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/modules/zfs/../../cddl/contrib/open=
 solaris/uts/common/fs/zfs/vdev_queue.c:489<br>#12 0xc0cb8119 in vdev_queue_=
 io (zio=3D0xc486c5b8)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/modules/z=
 fs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:768<br>#13=
  0xc0cd8704 in zio_vdev_io_start (ziop=3D&lt=3Bvalue optimized out&gt=3B)<b=
 r>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/modules/zfs/../../cddl/contrib/o=
 pensolaris/uts/common/fs/zfs/zio.c:2617<br>#14 0xc0cd4e28 in zio_execute (z=
 io=3D0xc486c5b8)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/modules/zfs/..=
 /../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1357<br>#15 0xc0cb7604=
  in vdev_mirror_io_start (zio=3D0xc476f000)<br>&nbsp=3B&nbsp=3B&nbsp=3B at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/v=
 dev_mirror.c:284<br>#16 0xc0cd854f in zio_vdev_io_start (ziop=3D&lt=3Bvalue=
  optimized out&gt=3B)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/modules/z=
 fs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2537<br>#17 0xc0c=
 d4e28 in zio_execute (zio=3D0xc476f000)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr=
 /src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c=
 :1357<br>#18 0xc0ca0ddc in spa_load_verify_cb (zilog=3D0x0=2C bp=3D&lt=3Bva=
 lue optimized out&gt=3B=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B dnp=3D0xc4b0fa00=2C =
 arg=3D&lt=3Bvalue optimized out&gt=3B)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/=
 src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:=
 1887<br>#19 0xc0c5eb79 in traverse_visitbp (td=3D0xd93f1120=2C dnp=3D0xc46a=
 b300=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B bp=3D0xc4b18100=2C zb=3D0xd93f06b0)<br>=
 &nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys/modules/zfs/../../cddl/contrib/ope=
 nsolaris/uts/common/fs/zfs/dmu_traverse.c:272<br>#20 0xc0c5edb0 in traverse=
 _visitbp (td=3D0xd93f1120=2C dnp=3D0xc4b0fa00=2C bp=3D0x80=2C<br>&nbsp=3B&n=
 bsp=3B&nbsp=3B zb=3D0xd93f07a8)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/src/sys=
 /modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.=
 c:306<br><br>failing line 489 is:<br><br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&n=
 bsp=3B&nbsp=3B&nbsp=3B if (avl_numnodes(&amp=3Bvq-&gt=3Bvq_active_tree) &gt=
 =3B=3D zfs_vdev_max_active)<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbs=
 p=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=
 =3B return (ZIO_PRIORITY_NUM_QUEUEABLE)=3B<br><br><br> 		 	   		  </div></b=
 ody>
 </html>=
 
 --_3fd876d8-95d2-45ec-9c06-a4c1351e561f_--

From: "Steven Hartland" <killing@multiplay.co.uk>
To: "Radim Kolar" <hsn@sendmail.cz>,
	"freebsd-fs@FreeBSD.org" <freebsd-fs@freebsd.org>,
	<bug-followup@freebsd.org>
Cc:  
Subject: Re: kern/189355: zfs panic on 10-stable
Date: Thu, 15 May 2014 19:59:28 +0100

 ----- Original Message ----- 
 From: "Radim Kolar" <hsn@sendmail.cz>
 failing line 489 is:
 
         if (avl_numnodes(&vq->vq_active_tree) >= zfs_vdev_max_active)
                 return (ZIO_PRIORITY_NUM_QUEUEABLE);
 
 Ok so thats what I thought it was could you see what vq is?
 
 
      

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "freebsd-fs@FreeBSD.org"
	<freebsd-fs@freebsd.org>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: zfs panic on 10-stable
Date: Thu, 15 May 2014 20:37:38 +0000

 --_8e795f21-165a-4c0e-8324-2a5a181326d7_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 > Ok so thats what I thought it was could you see what vq is?
 (kgdb) up 11
 #11 0xc0cb8187 in vdev_queue_io_to_issue (vq=3D0xc46ab300)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/f=
 s/zfs/vdev_queue.c:489
 489             if (avl_numnodes(&vq->vq_active_tree) >=3D zfs_vdev_max_act=
 ive)
 Current language:  auto=3B currently minimal
 (kgdb) print vq
 $1 =3D (vdev_queue_t *) 0xc46ab300
 (kgdb) print *vq
 $2 =3D {vq_vdev =3D 0xc46ab000=2C vq_class =3D {{vqc_active =3D 0=2C vqc_qu=
 eued_tree =3D {
         avl_root =3D 0x0=2C
         avl_compar =3D 0xc0cb7de0 <vdev_queue_timestamp_compare>=2C
         avl_offset =3D 476=2C avl_numnodes =3D 0=2C avl_size =3D 732}}=2C {
       vqc_active =3D 0=2C vqc_queued_tree =3D {avl_root =3D 0x0=2C
         avl_compar =3D 0xc0cb7de0 <vdev_queue_timestamp_compare>=2C
         avl_offset =3D 476=2C avl_numnodes =3D 0=2C avl_size =3D 732}}=2C {
       vqc_active =3D 1=2C vqc_queued_tree =3D {avl_root =3D 0x0=2C
         avl_compar =3D 0xc0cb7d70 <vdev_queue_offset_compare>=2C
         avl_offset =3D 476=2C avl_numnodes =3D 0=2C avl_size =3D 732}}=2C {
       vqc_active =3D 0=2C vqc_queued_tree =3D {avl_root =3D 0x0=2C
         avl_compar =3D 0xc0cb7d70 <vdev_queue_offset_compare>=2C
         avl_offset =3D 476=2C avl_numnodes =3D 0=2C avl_size =3D 732}}=2C {
       vqc_active =3D 0=2C vqc_queued_tree =3D {avl_root =3D 0xc486c794=2C
         avl_compar =3D 0xc0cb7d70 <vdev_queue_offset_compare>=2C
         avl_offset =3D 476=2C avl_numnodes =3D 1=2C avl_size =3D 732}}=2C {
       vqc_active =3D 0=2C vqc_queued_tree =3D {avl_root =3D 0x0=2C
         avl_compar =3D 0xc0cb7d70 <vdev_queue_offset_compare>=2C
         avl_offset =3D 476=2C avl_numnodes =3D 0=2C avl_size =3D 732}}}=2C
   vq_active_tree =3D {avl_root =3D 0xc476fa70=2C
     avl_compar =3D 0xc0cb7d70 <vdev_queue_offset_compare>=2C avl_offset =3D=
  476=2C
     avl_numnodes =3D 1=2C avl_size =3D 732}=2C vq_last_offset =3D 594196633=
 6=2C
   vq_io_complete_ts =3D 7702783485=2C vq_lock =3D {lock_object =3D {
       lo_name =3D 0xc0d6b0ff "vq->vq_lock"=2C lo_flags =3D 40960000=2C lo_d=
 ata =3D 0=2C
       lo_witness =3D 0x0}=2C sx_lock =3D 3290710800}}
 (kgdb)
  		 	   		  =
 
 --_8e795f21-165a-4c0e-8324-2a5a181326d7_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>&gt=3B Ok so thats what I though=
 t it was could you see what vq is?<br>(kgdb) up 11<br>#11 0xc0cb8187 in vde=
 v_queue_io_to_issue (vq=3D0xc46ab300)<br>&nbsp=3B&nbsp=3B&nbsp=3B at /usr/s=
 rc/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_qu=
 eue.c:489<br>489&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nb=
 sp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B if (avl_numnodes(&amp=3Bvq-&gt=3Bvq_a=
 ctive_tree) &gt=3B=3D zfs_vdev_max_active)<br>Current language:&nbsp=3B aut=
 o=3B currently minimal<br>(kgdb) print vq<br>$1 =3D (vdev_queue_t *) 0xc46a=
 b300<br>(kgdb) print *vq<br>$2 =3D {vq_vdev =3D 0xc46ab000=2C vq_class =3D =
 {{vqc_active =3D 0=2C vqc_queued_tree =3D {<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbs=
 p=3B&nbsp=3B&nbsp=3B&nbsp=3B avl_root =3D 0x0=2C<br>&nbsp=3B&nbsp=3B&nbsp=
 =3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B avl_compar =3D 0xc0cb7de0 &lt=3Bvdev_qu=
 eue_timestamp_compare&gt=3B=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&=
 nbsp=3B&nbsp=3B avl_offset =3D 476=2C avl_numnodes =3D 0=2C avl_size =3D 73=
 2}}=2C {<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B vqc_active =3D 0=2C vq=
 c_queued_tree =3D {avl_root =3D 0x0=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&=
 nbsp=3B&nbsp=3B&nbsp=3B avl_compar =3D 0xc0cb7de0 &lt=3Bvdev_queue_timestam=
 p_compare&gt=3B=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=
 =3B avl_offset =3D 476=2C avl_numnodes =3D 0=2C avl_size =3D 732}}=2C {<br>=
 &nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B vqc_active =3D 1=2C vqc_queued_tre=
 e =3D {avl_root =3D 0x0=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=
 =3B&nbsp=3B avl_compar =3D 0xc0cb7d70 &lt=3Bvdev_queue_offset_compare&gt=3B=
 =2C<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B avl_offset =
 =3D 476=2C avl_numnodes =3D 0=2C avl_size =3D 732}}=2C {<br>&nbsp=3B&nbsp=
 =3B&nbsp=3B&nbsp=3B&nbsp=3B vqc_active =3D 0=2C vqc_queued_tree =3D {avl_ro=
 ot =3D 0x0=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B a=
 vl_compar =3D 0xc0cb7d70 &lt=3Bvdev_queue_offset_compare&gt=3B=2C<br>&nbsp=
 =3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B avl_offset =3D 476=2C a=
 vl_numnodes =3D 0=2C avl_size =3D 732}}=2C {<br>&nbsp=3B&nbsp=3B&nbsp=3B&nb=
 sp=3B&nbsp=3B vqc_active =3D 0=2C vqc_queued_tree =3D {avl_root =3D 0xc486c=
 794=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B avl_comp=
 ar =3D 0xc0cb7d70 &lt=3Bvdev_queue_offset_compare&gt=3B=2C<br>&nbsp=3B&nbsp=
 =3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B avl_offset =3D 476=2C avl_numno=
 des =3D 1=2C avl_size =3D 732}}=2C {<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nb=
 sp=3B vqc_active =3D 0=2C vqc_queued_tree =3D {avl_root =3D 0x0=2C<br>&nbsp=
 =3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B avl_compar =3D 0xc0cb7d=
 70 &lt=3Bvdev_queue_offset_compare&gt=3B=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbs=
 p=3B&nbsp=3B&nbsp=3B&nbsp=3B avl_offset =3D 476=2C avl_numnodes =3D 0=2C av=
 l_size =3D 732}}}=2C<br>&nbsp=3B vq_active_tree =3D {avl_root =3D 0xc476fa7=
 0=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B avl_compar =3D 0xc0cb7d70 &lt=3Bvdev_queue=
 _offset_compare&gt=3B=2C avl_offset =3D 476=2C<br>&nbsp=3B&nbsp=3B&nbsp=3B =
 avl_numnodes =3D 1=2C avl_size =3D 732}=2C vq_last_offset =3D 5941966336=2C=
 <br>&nbsp=3B vq_io_complete_ts =3D 7702783485=2C vq_lock =3D {lock_object =
 =3D {<br>&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B lo_name =3D 0xc0d6b0ff "v=
 q-&gt=3Bvq_lock"=2C lo_flags =3D 40960000=2C lo_data =3D 0=2C<br>&nbsp=3B&n=
 bsp=3B&nbsp=3B&nbsp=3B&nbsp=3B lo_witness =3D 0x0}=2C sx_lock =3D 329071080=
 0}}<br>(kgdb)<br> 		 	   		  </div></body>
 </html>=
 
 --_8e795f21-165a-4c0e-8324-2a5a181326d7_--

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "bug-followup@FreeBSD.org"
	<bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Fri, 16 May 2014 07:31:54 +0000

 --_4a42f3dc-e58e-4892-96bd-2ccfd3354b76_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 line number will be probably wrong=2C i had such cases with clang in 10.0 a=
 lready if optimization is used. I am rebuilding kernel without optimalizati=
 on now.
  		 	   		  =
 
 --_4a42f3dc-e58e-4892-96bd-2ccfd3354b76_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>line number will be probably wro=
 ng=2C i had such cases with clang in 10.0 already if optimization is used. =
 I am rebuilding kernel without optimalization now.<br> 		 	   		  </div></b=
 ody>
 </html>=
 
 --_4a42f3dc-e58e-4892-96bd-2ccfd3354b76_--

From: "Steven Hartland" <smh@freebsd.org>
To: <bug-followup@freebsd.org>,
	"Radim Kolar" <hsn@sendmail.cz>
Cc:  
Subject: Re: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Fri, 16 May 2014 16:12:53 +0100

 Seems this didn't get through first time so resending.
 
 Hmm now that is odd as vq->vq_active_tree looks quite sane with
 avl_numnodes = 1, so theirs one outstanding zio in the active tree. 
 
 What might be of interest here is the stack indicates your zio is 
 a scrubbing read so there is something not quite right on the pool. 
 
 Is there some where you can tar up: 
 1. The kernel dump 
 2. Your current sources 
 3. Your current kernel 
 
 Then email me link privately to smh at freebsd.org, so I can have a
 look in more detail? 
 

From: Radim Kolar <hsn@sendmail.cz>
To: "freebsd-fs@FreeBSD.org" <freebsd-fs@freebsd.org>,
	"bug-followup@freebsd.org" <bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: zfs panic on 10-stable
Date: Thu, 22 May 2014 12:53:34 +0000

 --_d28ea596-b6a3-4db2-994e-4a7a36055ba8_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 i have troubles to compile kernel with optimalization disabled to get accur=
 ate line where it crashes. i run into two problems while trying to compile =
 ZFS without optimalization.
 =20
 kern/190101
 kern/190103 		 	   		  =
 
 --_d28ea596-b6a3-4db2-994e-4a7a36055ba8_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'><pre>i have troubles to compile =
 kernel with optimalization disabled to get accurate line where it crashes. =
 i run into two problems while trying to compile ZFS without optimalization.=
 <br> <br>kern/190101<br>kern/190103</pre> 		 	   		  </div></body>
 </html>=
 
 --_d28ea596-b6a3-4db2-994e-4a7a36055ba8_--

From: Radim Kolar <hsn@sendmail.cz>
To: "bug-followup@freebsd.org" <bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Thu, 22 May 2014 14:16:25 +0000

 --_5eaafcec-1bfd-4fa0-b05a-71b1e2909499_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 for google and other freebsd users:
 
 To compile freebsd kernel and its modules without optimization use
 
 makeoptions     DEBUG=3D"-g -O0"
 
 in kernel config file.
  		 	   		  =
 
 --_5eaafcec-1bfd-4fa0-b05a-71b1e2909499_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>for google and other freebsd use=
 rs:<br><br>To compile freebsd kernel and its modules without optimization u=
 se<br><br><b>makeoptions&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B DEBUG=3D"-g -O0"</=
 b><br><br>in kernel config file.<br> 		 	   		  </div></body>
 </html>=
 
 --_5eaafcec-1bfd-4fa0-b05a-71b1e2909499_--

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <smh@freebsd.org>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Thu, 22 May 2014 16:24:48 +0000

 --_0f2b99c2-5ea7-4f3d-ba3f-f126c437da89_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 after recompiling kernel and zfs with -O0=2C i am have very different stack=
 trace. I cant get dump with your patch in this case. Can you identify which=
  kernel component failed? its MPT driver?
 
 panic
 dblfault_handler
 cpu_search
 cpu_search_lowest
 sched_lowest
 sched_pickcpu
 sched_add
 intr_event_schedule_thread
 intr_event_handle
 intr_execute_handlers
 lapic_handle_intr
 Xapis_isr1
 bus_space_write_4
 mpt_write
 mpt_send_cmd
 mpt_execute_req
 bus_dmamap_load_ccb
 mpt_start
 mpt_action
 xpt_run_devq
 xpt_action_default
 scsi_action
 xpt_action
 dastart
 xpt_run_allocq
 xpt_schedule
 dareprobe
 g_disk_access
 g_access
 g_part_access
 g_access
 vdev_geom_attach_taster
 m_attach_taster
 vdev_geom_read_pool_label
 spa_generate_rootconf
 spa_import_rootpool
 zfs_mount
 vfs_domount_first
 vfs_domount
 vfs_donmount
 kernel_mount
 parse_mount
 vfs_mountroot_parse
  		 	   		  =
 
 --_0f2b99c2-5ea7-4f3d-ba3f-f126c437da89_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>after recompiling kernel and zfs=
  with -O0=2C i am have very different stacktrace. I cant get dump with your=
  patch in this case. Can you identify which kernel component failed? its MP=
 T driver?<br><br>panic<br>dblfault_handler<br>cpu_search<br>cpu_search_lowe=
 st<br>sched_lowest<br>sched_pickcpu<br>sched_add<br>intr_event_schedule_thr=
 ead<br>intr_event_handle<br>intr_execute_handlers<br>lapic_handle_intr<br>X=
 apis_isr1<br>bus_space_write_4<br>mpt_write<br>mpt_send_cmd<br>mpt_execute_=
 req<br>bus_dmamap_load_ccb<br>mpt_start<br>mpt_action<br>xpt_run_devq<br>xp=
 t_action_default<br>scsi_action<br>xpt_action<br>dastart<br>xpt_run_allocq<=
 br>xpt_schedule<br>dareprobe<br>g_disk_access<br>g_access<br>g_part_access<=
 br>g_access<br>vdev_geom_attach_taster<br>m_attach_taster<br>vdev_geom_read=
 _pool_label<br>spa_generate_rootconf<br>spa_import_rootpool<br>zfs_mount<br=
 >vfs_domount_first<br>vfs_domount<br>vfs_donmount<br>kernel_mount<br>parse_=
 mount<br>vfs_mountroot_parse<br> 		 	   		  </div></body>
 </html>=
 
 --_0f2b99c2-5ea7-4f3d-ba3f-f126c437da89_--

From: Andriy Gapon <avg@FreeBSD.org>
To: bug-followup@FreeBSD.org, hsn@sendmail.cz
Cc:  
Subject: Re: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Thu, 22 May 2014 20:54:02 +0300

 This looks like a possible stack exhaustion.
 
 -- 
 Andriy Gapon

From: "Steven Hartland" <killing@multiplay.co.uk>
To: <bug-followup@freebsd.org>,
	"Radim Kolar" <hsn@sendmail.cz>
Cc:  
Subject: Re: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Thu, 22 May 2014 19:51:28 +0100

 Silly question are you using i386 and not amd64 architecture?
 
 If so can your try adding the following to your kernel config:
 options KSTACK_PAGES=4
 
     Regards
     Steve

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>
Cc:  
Subject: RE: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Fri, 23 May 2014 10:30:08 +0000

 --_b76a0af4-5d00-49ea-9642-a6f444716426_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 yes its i386=2C i will recompile kernel and report results.
  		 	   		  =
 
 --_b76a0af4-5d00-49ea-9642-a6f444716426_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>yes its i386=2C i will recompile=
  kernel and report results.<br> 		 	   		  </div></body>
 </html>=
 
 --_b76a0af4-5d00-49ea-9642-a6f444716426_--

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>
Cc:  
Subject: RE: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Fri, 23 May 2014 16:26:13 +0000

 --_7507ba2e-9b9b-4475-96e5-477aa8a382b7_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 with "options KSTACK_PAGES=3D4" in kernel config it do not panic anymore. P=
 lease commit patch for making it default value on i386 (32-bit)
  		 	   		  =
 
 --_7507ba2e-9b9b-4475-96e5-477aa8a382b7_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>with "options KSTACK_PAGES=3D4" =
 in kernel config it do not panic anymore. Please commit patch for making it=
  default value on i386 (32-bit)<br> 		 	   		  </div></body>
 </html>=
 
 --_7507ba2e-9b9b-4475-96e5-477aa8a382b7_--

From: "Steven Hartland" <killing@multiplay.co.uk>
To: "Radim Kolar" <hsn@sendmail.cz>,
	<bug-followup@freebsd.org>,
	<freebsd-fs@freebsd.org>
Cc:  
Subject: Re: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Fri, 23 May 2014 17:57:17 +0100

 This is already noted in UPDATING.
 
 With i386 being an aging tech which really shouldn't really be used to run ZFS
 and that has machines with often small amounts of memory this should be left
 to those who do require it to build a custom kernel I'm afraid.

From: Radim Kolar <hsn@sendmail.cz>
To: Steven Hartland <killing@multiplay.co.uk>, "bug-followup@freebsd.org"
	<bug-followup@freebsd.org>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>
Cc:  
Subject: RE: kern/189355: [zfs] zfs panic on root mount 10-stable
Date: Fri, 23 May 2014 17:23:03 +0000

 --_db38143b-2d44-4957-bd01-080e9fe18e82_
 Content-Type: text/plain; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 its not just about building custom kernel. GENERIC from 10-STABLE panics on=
  i386 mount ZFS root too.
 
 Maybe ZFS should not be used on i386=2C but it should not panic system. If =
 you are against changing default kernel configuration on i386 then add anot=
 her warning to boot messages similar to this:
 
 ZFS WARNING: Recommended minimum kmem_size is 512MB=3B expect unstable beha=
 vior.
              Consider tuning vm.kmem_size and vm.kmem_size_max
              in /boot/loader.conf.
  		 	   		  =
 
 --_db38143b-2d44-4957-bd01-080e9fe18e82_
 Content-Type: text/html; charset="iso-8859-2"
 Content-Transfer-Encoding: quoted-printable
 
 <html>
 <head>
 <style><!--
 .hmmessage P
 {
 margin:0px=3B
 padding:0px
 }
 body.hmmessage
 {
 font-size: 12pt=3B
 font-family:Calibri
 }
 --></style></head>
 <body class=3D'hmmessage'><div dir=3D'ltr'>its not just about building cust=
 om kernel. GENERIC from 10-STABLE panics on i386 mount ZFS root too.<br><br=
 >Maybe ZFS should not be used on i386=2C but it should not panic system. If=
  you are against changing default kernel configuration on i386 then add ano=
 ther warning to boot messages similar to this:<br><br>ZFS WARNING: Recommen=
 ded minimum kmem_size is 512MB=3B expect unstable behavior.<br>&nbsp=3B&nbs=
 p=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=
 =3B&nbsp=3B Consider tuning vm.kmem_size and vm.kmem_size_max<br>&nbsp=3B&n=
 bsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=3B&nbsp=
 =3B&nbsp=3B in /boot/loader.conf.<br> 		 	   		  </div></body>
 </html>=
 
 --_db38143b-2d44-4957-bd01-080e9fe18e82_--
>Unformatted:
