From nobody@FreeBSD.org  Fri Apr 25 19:16:20 2014
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id 4430DB31
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 25 Apr 2014 19:16:20 +0000 (UTC)
Received: from cgiserv.freebsd.org (cgiserv.freebsd.org [IPv6:2001:1900:2254:206a::50:4])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 3142C1B73
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 25 Apr 2014 19:16:20 +0000 (UTC)
Received: from cgiserv.freebsd.org ([127.0.1.6])
	by cgiserv.freebsd.org (8.14.8/8.14.8) with ESMTP id s3PJGKp7017431
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 25 Apr 2014 19:16:20 GMT
	(envelope-from nobody@cgiserv.freebsd.org)
Received: (from nobody@localhost)
	by cgiserv.freebsd.org (8.14.8/8.14.8/Submit) id s3PJGKsv017425;
	Fri, 25 Apr 2014 19:16:20 GMT
	(envelope-from nobody)
Message-Id: <201404251916.s3PJGKsv017425@cgiserv.freebsd.org>
Date: Fri, 25 Apr 2014 19:16:20 GMT
From: Alan Somers <asomers@freebsd.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: lock order reversal: (sleepable after non-sleepable) in lacp_attach
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         188997
>Category:       kern
>Synopsis:       [lor] lock order reversal: (sleepable after non-sleepable) in lacp_attach
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    scottl
>State:          analyzed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Apr 25 19:20:00 UTC 2014
>Closed-Date:    
>Last-Modified:  Sun May 04 02:59:33 UTC 2014
>Originator:     Alan Somers
>Release:        11.0 CURRENT
>Organization:
Spectra Logic
>Environment:
FreeBSD alans-fbsd-head 11.0-CURRENT FreeBSD 11.0-CURRENT #49 r264887M: Thu Apr 24 17:21:48 MDT 2014 alans@ns1.eng.sldomain.com:/vmpool/obj/usr/home/alans freebsd/head/sys/GENERIC amd64
>Description:
r253687 introduced a lock order reversal.  It's a lock of a sleepable lock while a nonsleepable lock is still held.  The problem is that lacp_attach creates some sysctl nodes, but further up the stack lagg_ioctl called LAGG_WLOCK() on the softc.
>How-To-Repeat:
ifconfig tap0 create
ifconfig tap1 create
ifconfig tap2 create
ifconfig lagg0 create
ifconfig lagg0 up laggproto lacp laggport tap0 laggport tap1 laggport
tap2 192.0.0.2/24
>Fix:
Two possible solutions come to mind:

1) Move the sysctl node creation out of lacp_attach.  Call it from lagg_ioctl after LAGG_WUNLOCK()

2) Turn the sysctl nodes into ioctls instead.  That was proposed for a different reason at http://lists.freebsd.org/pipermail/svn-src-head/2013-July/049602.html .

>Release-Note:
>Audit-Trail:

From: asomers@gmail.com
To: bug-followup@FreeBSD.org, asomers@freebsd.org
Cc:  
Subject: Re: kern/188997: lock order reversal: (sleepable after non-sleepable)
 in lacp_attach
Date: Fri, 25 Apr 2014 13:42:27 -0600

 Forgot to include the stack trace:
 
 lock order reversal: (sleepable after non-sleepable)
 1st 0xfffff8000252ca08 if_lagg rmlock (if_lagg rmlock)
 @/usr/home/alans/freebsd/head/sys/modules/if_lagg/../../net/if_lagg.c:1040
 2nd 0xffffffff814ef4e0 sysctl lock (sysctl lock)
 @/usr/home/alans/freebsd/head/sys/kern/kern_sysctl.c:474
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00977485b0
 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0097748660
 witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe00977486f0
 _sx_xlock() at _sx_xlock+0x75/frame 0xfffffe0097748730
 sysctl_add_oid() at sysctl_add_oid+0x4a/frame 0xfffffe0097748780
 lacp_attach() at lacp_attach+0xf7/frame 0xfffffe00977487f0
 lagg_lacp_attach() at lagg_lacp_attach+0x88/frame 0xfffffe0097748810
 lagg_ioctl() at lagg_ioctl+0x98a/frame 0xfffffe00977488f0
 in_control() at in_control+0x38e/frame 0xfffffe0097748970
 ifioctl() at ifioctl+0xba2/frame 0xfffffe0097748a30
 kern_ioctl() at kern_ioctl+0x22b/frame 0xfffffe0097748a90
 sys_ioctl() at sys_ioctl+0x13c/frame 0xfffffe0097748ae0
 amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe0097748bf0
 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0097748bf0
 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800fa045a, rsp
 =0x7fffffffe118, rbp = 0x7fffffffe1a0 ---
 uma_zalloc_arg: zone "128" with the following non-sleepable locks held:
 exclusive rm if_lagg rmlock (if_lagg rmlock) r = 0
 (0xfffff8000252ca08) locked
 @/usr/home/alans/freebsd/head/sys/modules/if_lagg/../../net/if_lagg.c:1040
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0097748500
 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00977485b0
 witness_warn() at witness_warn+0x4b5/frame 0xfffffe0097748670
 uma_zalloc_arg() at uma_zalloc_arg+0x3b/frame 0xfffffe00977486e0
 malloc() at malloc+0x194/frame 0xfffffe0097748730
 sysctl_add_oid() at sysctl_add_oid+0x11f/frame 0xfffffe0097748780
 lacp_attach() at lacp_attach+0xf7/frame 0xfffffe00977487f0
 lagg_lacp_attach() at lagg_lacp_attach+0x88/frame 0xfffffe0097748810
 lagg_ioctl() at lagg_ioctl+0x98a/frame 0xfffffe00977488f0
 in_control() at in_control+0x38e/frame 0xfffffe0097748970
 ifioctl() at ifioctl+0xba2/frame 0xfffffe0097748a30
 kern_ioctl() at kern_ioctl+0x22b/frame 0xfffffe0097748a90
 sys_ioctl() at sys_ioctl+0x13c/frame 0xfffffe0097748ae0
 amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe0097748bf0
 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0097748bf0
 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800fa045a, rsp =
 0x7fffffffe118, rbp = 0x7fffffffe1a0 ---
State-Changed-From-To: open->analyzed 
State-Changed-By: asomers 
State-Changed-When: Mon Apr 28 17:29:07 UTC 2014 
State-Changed-Why:  
Scott volunteered himself 


Responsible-Changed-From-To: freebsd-bugs->scottl 
Responsible-Changed-By: asomers 
Responsible-Changed-When: Mon Apr 28 17:29:07 UTC 2014 
Responsible-Changed-Why:  
Scott volunteered himself 

http://www.freebsd.org/cgi/query-pr.cgi?pr=188997 
>Unformatted:
