From nobody@FreeBSD.org  Fri Mar  2 15:26:30 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 8ADDE1065670
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  2 Mar 2012 15:26:30 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 6BD508FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Fri,  2 Mar 2012 15:26:30 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q22FQUKw087151
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 2 Mar 2012 15:26:30 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q22FQUmv087150;
	Fri, 2 Mar 2012 15:26:30 GMT
	(envelope-from nobody)
Message-Id: <201203021526.q22FQUmv087150@red.freebsd.org>
Date: Fri, 2 Mar 2012 15:26:30 GMT
From: Vladislav Movchan <vladislav.movchan@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [ndis][panic][patch] IRQL_NOT_GREATER_THAN
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         165630
>Category:       kern
>Synopsis:       [ndis][panic][patch] IRQL_NOT_GREATER_THAN
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          patched
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Mar 02 15:30:06 UTC 2012
>Closed-Date:    
>Last-Modified:  Wed Jul 03 00:52:04 UTC 2013
>Originator:     Vladislav Movchan
>Release:        FreeBSD 10.0-CURRENT
>Organization:
>Environment:
FreeBSD starlight 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r232379: Fri Mar 2 13:24:25 EET 2012 root@starlight:/usr/obj/usr/src/sys/Mephistopheles amd64
>Description:
There is a race condition inside of sys/compat/ndis/subr_hal.c:KfRaiseIrql() function which could lead to "IRQL_NOT_GREATER_THAN" panic.

Race could happen if after execution of this line:
oldirql = KeGetCurrentIrql();

kthread got preempted and later appeared on another CPU core which has another "irq level".
In this case "oldirql != DISPATCH_LEVEL" check will have no much sense as on current CPU core "irq level" could be different. As result we will skip sched_pin + mtx_lock lines when it was necessary execute them, or execute then when it necessary to skip.


To fix this race we pin to current CPU core before obtaining "irq level", then we made "oldirql != DISPATCH_LEVEL" check and later we "unpin" (if we find that previous sched_pin() was not actually necessary).


Panic message:
panic: IRQL_NOT_GREATER_THAN
cpuid = 3
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
panic() at panic+0x1cd
KfLowerIrql() at KfLowerIrql+0x62
ntoskrnl_workitem_thread() at ntoskrnl_workitem_thread+0xab
fork_exit() at fork_exit+0x189
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80d111acf0, rbp = 0 ---
KDB: enter: panic


Backtrace:
#0  doadump (textdump=-787372224) at /usr/src/sys/kern/kern_shutdown.c:268
268             if (textdump && textdump_pending) {
(kgdb) #0  doadump (textdump=-787372224) at /usr/src/sys/kern/kern_shutdown.c:268
#1  0xffffffff802de36c in db_fncall (dummy1=Variable "dummy1" is not available.
)
    at /usr/src/sys/ddb/db_command.c:573
#2  0xffffffff802de6a1 in db_command (last_cmdp=0xffffffff80dcd560, cmd_table=Variable "cmd_table" is not available.

) at /usr/src/sys/ddb/db_command.c:449
#3  0xffffffff802de8f0 in db_command_loop ()
    at /usr/src/sys/ddb/db_command.c:502
#4  0xffffffff802e0ad4 in db_trap (type=Variable "type" is not available.
) at /usr/src/sys/ddb/db_main.c:229
#5  0xffffffff806d3bb1 in kdb_trap (type=3, code=0, tf=0xffffff80d111a970)
    at /usr/src/sys/kern/subr_kdb.c:629
#6  0xffffffff80942f68 in trap (frame=0xffffff80d111a970)
    at /usr/src/sys/amd64/amd64/trap.c:591
#7  0xffffffff8092c1cf in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#8  0xffffffff806d36ab in kdb_enter (why=0xffffffff80a3fb16 "panic",
    msg=0x80 <Address 0x80 out of bounds>) at cpufunc.h:63
#9  0xffffffff8069aea6 in panic (fmt=Variable "fmt" is not available.
)
    at /usr/src/sys/kern/kern_shutdown.c:633
#10 0xffffffff840d6ff2 in KfLowerIrql (oldirql=Variable "oldirql" is not available.
)
    at /usr/src/sys/modules/ndis/../../compat/ndis/subr_hal.c:417
#11 0xffffffff840dba5b in ntoskrnl_workitem_thread (arg=Variable "arg" is not available.
)
    at /usr/src/sys/modules/ndis/../../compat/ndis/subr_ntoskrnl.c:2766
#12 0xffffffff8066b039 in fork_exit (
    callout=0xffffffff840db9b0 <ntoskrnl_workitem_thread>,
    arg=0xfffffe00206a9cc0, frame=0xffffff80d111ac40)
    at /usr/src/sys/kern/kern_fork.c:992
#13 0xffffffff8092c6fe in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:602
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000001 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0xffffffff8240f200 in tdq_cpu ()
#39 0x0000000000000000 in ?? ()
#40 0xffffffff8240f1c0 in affinity ()
#41 0xfffffe0020971900 in ?? ()
#42 0xffffff80d111a980 in ?? ()
#43 0xffffff80d111a928 in ?? ()
#44 0xfffffe00082bf900 in ?? ()
#45 0xffffffff806c64ac in sched_switch (td=0xfffffe00206a9cc0,
    newtd=0xffffffff840db9b0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1890
Previous frame inner to this frame (corrupt stack?)
(kgdb)


>How-To-Repeat:
There are no special conditions necessary to reproduce this panic - just load ndis interface with traffic and wait for the race.
I've used benchmarks/iperf running 10 parallel threads ("-P 10" flag) with tcp window set to 1Mb ("-w 1m" flag). Machine with ndis interface was a "client" of iperf (so it was mostly sending). 

Adding bpf(4) consumers (trafshow or tcpdump) allows to trigger this race/panic faster.
Using faster NIC (1Gbps NIC instead of wifi) allows to trigger this race/panic faster.
>Fix:
Attached patch fixed this problem for me.

Patch attached with submission follows:

Index: /usr/src/sys/compat/ndis/subr_hal.c
===================================================================
--- /usr/src/sys/compat/ndis/subr_hal.c	(revision 232379)
+++ /usr/src/sys/compat/ndis/subr_hal.c	(working copy)
@@ -392,16 +392,17 @@
 {
 	uint8_t			oldirql;
 
+	sched_pin();
 	oldirql = KeGetCurrentIrql();
 
 	/* I am so going to hell for this. */
 	if (oldirql > irql)
 		panic("IRQL_NOT_LESS_THAN");
 
-	if (oldirql != DISPATCH_LEVEL) {
-		sched_pin();
+	if (oldirql != DISPATCH_LEVEL)
 		mtx_lock(&disp_lock[curthread->td_oncpu]);
-	}
+	else
+		sched_unpin();
 /*printf("RAISE IRQL: %d %d\n", irql, oldirql);*/
 
 	return (oldirql);


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->brucec 
Responsible-Changed-By: brucec 
Responsible-Changed-When: Sat Mar 3 17:36:46 UTC 2012 
Responsible-Changed-Why:  
Take. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=165630 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/165630: commit references a PR
Date: Sun,  4 Mar 2012 17:09:26 +0000 (UTC)

 Author: brucec
 Date: Sun Mar  4 17:08:43 2012
 New Revision: 232509
 URL: http://svn.freebsd.org/changeset/base/232509
 
 Log:
   Fix race condition in KfRaiseIrql().
   
   After getting the current irql, if the kthread gets preempted and
   subsequently runs on a different CPU, the saved irql could be wrong.
   
   Also, correct the panic string.
   
   PR:		kern/165630
   Submitted by:	Vladislav Movchan <vladislav.movchan at gmail.com>
 
 Modified:
   head/sys/compat/ndis/subr_hal.c
 
 Modified: head/sys/compat/ndis/subr_hal.c
 ==============================================================================
 --- head/sys/compat/ndis/subr_hal.c	Sun Mar  4 17:00:46 2012	(r232508)
 +++ head/sys/compat/ndis/subr_hal.c	Sun Mar  4 17:08:43 2012	(r232509)
 @@ -392,16 +392,18 @@ KfRaiseIrql(uint8_t irql)
  {
  	uint8_t			oldirql;
  
 +	sched_pin();
  	oldirql = KeGetCurrentIrql();
  
  	/* I am so going to hell for this. */
  	if (oldirql > irql)
 -		panic("IRQL_NOT_LESS_THAN");
 +		panic("IRQL_NOT_LESS_THAN_OR_EQUAL");
  
 -	if (oldirql != DISPATCH_LEVEL) {
 -		sched_pin();
 +	if (oldirql != DISPATCH_LEVEL) 
  		mtx_lock(&disp_lock[curthread->td_oncpu]);
 -	}
 +	else
 +		sched_unpin();	
 +
  /*printf("RAISE IRQL: %d %d\n", irql, oldirql);*/
  
  	return (oldirql);
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: brucec 
State-Changed-When: Sun Mar 4 17:10:42 UTC 2012 
State-Changed-Why:  
Fixed in r232509. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=165630 

From: Glen Barber <gjb@FreeBSD.org>
To: brucec@FreeBSD.org
Cc: bug-followup@FreeBSD.org, vladislav.movchan@gmail.com
Subject: Re: kern/165630: [ndis][panic][patch] IRQL_NOT_GREATER_THAN
Date: Wed, 12 Sep 2012 22:06:23 -0400

 --mJm6k4Vb/yFcL9ZU
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 Bruce,
 
 Any chance you can MFC r232509, changed for this PR?
 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/165630
 
 Glen
 
 
 --mJm6k4Vb/yFcL9ZU
 Content-Type: application/pgp-signature
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.19 (FreeBSD)
 
 iQEcBAEBCAAGBQJQUT+fAAoJEFJPDDeguUajZO0H/iBppJrz4AImqf44vLhTw7zn
 HLFBJeC9+A6jI4zLFplGFXGpCy5/BauZDxrOUMV8hlpj2tfby/evt8DItbDxk4Wu
 698m+3kHoQB3s7NLaba6E2fLMCC9f1EiItmUOhgXwrJKXdPjYy2ZzGgCsUvvJ+6l
 cqeGfQt7qq8Agap7pQ9IgPRwji+3mha7WhqsU7ahd20H/uJpyJaRhPnBD8XsOg2S
 CtVdAF8UY7WDBkEvEKV+8trxCcryi6SoR+UJL7bACx9V9XI/KimwbFvdY3OGNK1D
 acM6AHjyZP5iUXU1U2nGE6tQAqkhH3EeSfc0emTqollpONo7rmlqB3fSKcE2jeo=
 =jIKL
 -----END PGP SIGNATURE-----
 
 --mJm6k4Vb/yFcL9ZU--
Responsible-Changed-From-To: brucec->freebsd-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Jul 3 00:50:32 UTC 2013 
Responsible-Changed-Why:  
commit bit has been taken in for safekeeping. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=165630 
>Unformatted:
