From nobody@FreeBSD.org  Thu Dec  1 15:18:30 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id BD2941065673
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  1 Dec 2011 15:18:30 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 935918FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  1 Dec 2011 15:18:30 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id pB1FIToa025496
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 1 Dec 2011 15:18:29 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id pB1FITWs025495;
	Thu, 1 Dec 2011 15:18:29 GMT
	(envelope-from nobody)
Message-Id: <201112011518.pB1FITWs025495@red.freebsd.org>
Date: Thu, 1 Dec 2011 15:18:29 GMT
From: "KAHO, Toshikazu" <kaho@elam.kais.kyoto-u.ac.jp>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [GEOM]multiple gmirror cause kernel panic during shutdown
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         162997
>Category:       kern
>Synopsis:       [geom] multiple gmirror cause kernel panic during shutdown
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-geom
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Dec 01 15:20:14 UTC 2011
>Closed-Date:    Mon Dec 05 08:45:52 UTC 2011
>Last-Modified:  Mon Dec 05 08:45:52 UTC 2011
>Originator:     KAHO, Toshikazu
>Release:        10.0-CURRENT
>Organization:
>Environment:
FreeBSD pf2.ed.niigata-u.ac.jp 10.0-CURRENT FreeBSD 10.0-CURRENT #3 r227014:228180: Thu Dec  1 23:01:58 JST 2011     kaho@pf2.ed.niigata-u.ac.jp:/usr/obj/usr/src/sys/NUPA  amd64
>Description:
r227015(sys/geom/geom_vfs.c) causes kernel panic at destroying geom providers.
A system with only one gmirror provider does not affect this problem.
The machines with mirroring slice level and having multiple gmirror names
panic during shutdown. It occurs both i386 and amd64 system.
i386 system shows:

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x10
fault code              = supervisor write, page not present
instruction pointer     = 0x20:0xc052e601
stack pointer           = 0x28:0xc1b0ec80
frame pointer           = 0x28:0xc1b0eca0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 13 (g_event)

(kgdb) bt
#0  doadump (textdump=0) at pcpu.h:244
#1  0xc04a2bb3 in db_dump (dummy=-1068308991, dummy2=0, dummy3=-1, 
    dummy4=0xc1b0ea0c "") at /usr/src/sys/ddb/db_command.c:537
#2  0xc04a25f1 in db_command (last_cmdp=0xc08b003c, cmd_table=0x0, dopager=1)
    at /usr/src/sys/ddb/db_command.c:448
#3  0xc04a2755 in db_command_loop () at /usr/src/sys/ddb/db_command.c:501
#4  0xc04a47dc in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_main.c:229
#5  0xc05c1bfd in kdb_trap (type=12, code=0, tf=0xc1b0ec40)
    at /usr/src/sys/kern/subr_kdb.c:625
#6  0xc07e6f9f in trap_fatal (frame=0xc1b0ec40, eva=16)
    at /usr/src/sys/i386/i386/trap.c:966
#7  0xc07e7099 in trap_pfault (frame=0xc1b0ec40, usermode=0, eva=16)
    at /usr/src/sys/i386/i386/trap.c:839
#8  0xc07e7ec7 in trap (frame=0xc1b0ec40) at /usr/src/sys/i386/i386/trap.c:558
#9  0xc07d3a6c in calltrap () at /usr/src/sys/i386/i386/exception.s:168
#10 0xc052e601 in g_vfs_orphan (cp=0xc1e79440) at atomic.h:246
#11 0xc052905d in g_run_events () at /usr/src/sys/geom/geom_event.c:211
#12 0xc052a648 in g_event_procbody (arg=0x0)
    at /usr/src/sys/geom/geom_kern.c:122
#13 0xc05631a2 in fork_exit (callout=0xc052a5e0 <g_event_procbody>, arg=0x0, 
    frame=0xc1b0ed28) at /usr/src/sys/kern/kern_fork.c:995
#14 0xc07d3ae4 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:275

>How-To-Repeat:
setup more than 2 gmirror names and shutdown
>Fix:
I don't have a solution but 
kernel with `svn sys/geom/geom_vfs.c -r 227014` does not panic.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-geom 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Thu Dec 1 16:49:44 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=162997 

From: Florian Smeets <flo@FreeBSD.org>
To: bug-followup@FreeBSD.org, kaho@elam.kais.kyoto-u.ac.jp,
        Alexander Motin <mav@FreeBSD.org>
Cc:  
Subject: Re: kern/162997: [geom] multiple gmirror cause kernel panic during
 shutdown
Date: Thu, 01 Dec 2011 19:37:40 +0100

 This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
 --------------enigDF0133215EFDC8445D960449
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 I was tracking down a similar problem. My sparc64 machine with multiple
 gmirrors stopped rebooting, it hangs after the first mirror is
 destroyed, and never recovers. I have to reset it via LOM.
 
 I can confirm the hang is caused by r227015.
 
 
 Syncing disks, vnodes remaining...1 1 0 0 done
 GEOM_MIRROR: Device var: provider mirror/var destroyed.
 GEOM_MIRROR: Device var destroyed.
 *hang*
 
 Florian
 
 
 --------------enigDF0133215EFDC8445D960449
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 
 iEYEARECAAYFAk7XyXQACgkQapo8P8lCvwl8mwCgvHS+LG43lEQcVF8g42DKdkb4
 vyMAoIShG99Yk389ygv+wtlZogurjUbt
 =ALRN
 -----END PGP SIGNATURE-----
 
 --------------enigDF0133215EFDC8445D960449--

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/162997: commit references a PR
Date: Fri,  2 Dec 2011 17:10:11 +0000 (UTC)

 Author: mav
 Date: Fri Dec  2 17:09:48 2011
 New Revision: 228204
 URL: http://svn.freebsd.org/changeset/base/228204
 
 Log:
   Close race between geom destruction on g_vfs_close() when softc destroyed
   and g_vfs_orphan() call that tries to access softc, intruced at r227015.
   
   PR:		kern/162997
 
 Modified:
   head/sys/geom/geom_vfs.c
 
 Modified: head/sys/geom/geom_vfs.c
 ==============================================================================
 --- head/sys/geom/geom_vfs.c	Fri Dec  2 15:47:05 2011	(r228203)
 +++ head/sys/geom/geom_vfs.c	Fri Dec  2 17:09:48 2011	(r228204)
 @@ -169,8 +169,10 @@ g_vfs_orphan(struct g_consumer *cp)
  	g_topology_assert();
  
  	gp = cp->geom;
 -	sc = gp->softc;
  	g_trace(G_T_TOPOLOGY, "g_vfs_orphan(%p(%s))", cp, gp->name);
 +	sc = gp->softc;
 +	if (sc == NULL)
 +		return;
  	mtx_lock(&sc->sc_mtx);
  	sc->sc_orphaned = 1;
  	destroy = (sc->sc_active == 0);
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: Kaho Toshikazu <kaho@elam.kais.kyoto-u.ac.jp>
To: Florian Smeets <flo@freebsd.org>
Cc: bug-followup@freebsd.org, Alexander Motin <mav@freebsd.org>
Subject: Re: kern/162997: [geom] multiple gmirror cause kernel panic during shutdown
Date: Mon, 05 Dec 2011 10:06:11 +0900

 Hello,
 
 I don't like to crash real machine many times and build test
 environment on qemu. md0p2a is labeled gm0 and md0p2h is gm1,
 and mount as UFS2. After sysctl kern.geom.debugflags=7,
 machine was rebooted. The copy from console before panic is here.
 
 open delta:[r-1w-1e-3] old:[r2w2e6] provider:[r2w2e6] 0xc14eac00(md0)
 g_post_event_x(0xc052c830, 0xc166c300, 2, 0)
   ref 0xc166c300
 g_post_event_x(0xc0a03e40, 0xc1446b00, 2, 0)
 g_wither_geom(0xc17ffa80(gm1.sync))
 GEOM_MIRROR: Device gm1 destroyed.
 g_wither_geom(0xc17ffb00(gm1))
 g_orphan_register(mirror/gm1)
 g_vfs_orphan(0xc1800400(ffs.mirror/gm1))
 kernel trap 12 with interrupts disabled
 
 The situation looks like this:
 gm1 was destroyed in g_vfs_close() and then g_vfs_orphan() was called to
 manipulate gm1. The function g_vfs_close() was freed softc and 
 g_vfs_orphan() would like to use softc already freed and it causes
 panic. 
 
 I think that malloc() in g_vfs_open() and free() in g_vfs_close()
 for mtx_lock is not valid method. malloc() should not be used,
 or fee() should be used in other function. Or correct other code
 which never call destroyed provider.
 
 --
 kaho Toshikazu

From: Kaho Toshikazu <kaho@elam.kais.kyoto-u.ac.jp>
To: bug-followup@FreeBSD.org
Cc: mav@FreeBSD.org
Subject: Re: kern/162997: [geom] multiple gmirror cause kernel panic during shutdown
Date: Mon, 05 Dec 2011 12:33:10 +0900

 I missed r228204 and it makes machine reboot without panic.
 Thanks.
 
 -- 
 Kaho Toshikazu
State-Changed-From-To: open->closed 
State-Changed-By: mav 
State-Changed-When: Mon Dec 5 08:45:17 UTC 2011 
State-Changed-Why:  
Problem fixed by r228204. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=162997 
>Unformatted:
