From nobody@FreeBSD.org  Thu Dec  6 14:50:06 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id A95762A9
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  6 Dec 2012 14:50:06 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 8F9DB8FC19
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  6 Dec 2012 14:50:06 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id qB6Eo5kG046263
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 6 Dec 2012 14:50:05 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id qB6Eo5d3046262;
	Thu, 6 Dec 2012 14:50:05 GMT
	(envelope-from nobody)
Message-Id: <201212061450.qB6Eo5d3046262@red.freebsd.org>
Date: Thu, 6 Dec 2012 14:50:05 GMT
From: Lutz Donnerhacke <lutz@donnerhacke.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: "kldunload ipdivert" corrupts kernel
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         174236
>Category:       kern
>Synopsis:       [modules] "kldunload ipdivert" corrupts kernel
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Dec 06 15:00:00 UTC 2012
>Closed-Date:    
>Last-Modified:  Sun Dec 09 17:29:43 UTC 2012
>Originator:     Lutz Donnerhacke
>Release:        FreeBSD 8.3-RELEASE (GENERIC)
>Organization:
IKS Service GmbH
>Environment:
FreeBSD server6 8.3-RELEASE FreeBSD 8.3-RELEASE #0: Mon Apr  9 21:23:18 UTC 2012     root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:
After unsuccessful kldunload of ipdivert.ko, all kernel module maintainence operations block.
>How-To-Repeat:
Reboot the server to a fresh state. In short:
 # kldload ipfw
 # kldload ipdivert
 # kldunload ipdivert
 # kldstat
 {hangs]

And now the glory details:

[root@server6 ~]# cat /boot/loader.conf
aacu64_load="YES"
if_lagg_load="YES"
net.inet.ip.fw.default_to_accept=1
ipfw_load="YES"
ipfw_nat_load="YES"

[root@server6 ~]# grep firewall /etc/rc.conf
firewall_enable="YES"
firewall_type="/etc/firewall.rules"

[root@server6 ~]# cat /etc/firewall.rules
-f flush
# defining NAT pools
nat 128 config ip x.y.z.128 log same_ports
[... same up to ...]
nat 159 config ip x.y.z.159 log same_ports
# active ruleset
add   10 deny log ip4 from any to any not verrevpath in
add 1128 nat 128 ipv4 from 100.64.0.0:255.192.0.31 to any out
[... same up to ...]
add 1159 nat 159 ipv4 from 100.64.0.31:255.192.0.31 to any out
add 2128 nat 128 ip4 from any to x.y.z.128
[... same up to ...]
add 2159 nat 159 ip4 from any to x.y.z.159
# Diverting dhpcv6 in future
# add 9000 divert 12345 ipv6 from fe80::/16 to ff02::1:2 via ng* in
# Allow everything
add 9999 allow all from any to any

[root@server6 ~]# kldstat
Id Refs Address            Size     Name
 1   34 0xffffffff80100000 e56d68   kernel
 2    2 0xffffffff80f57000 1a3e0    ipfw.ko
 3    1 0xffffffff80f72000 192b0    aacu64.ko
 4    1 0xffffffff80f8c000 c470     if_lagg.ko
 5    1 0xffffffff80f99000 4088     ipfw_nat.ko
 6    2 0xffffffff80f9e000 14320    libalias.ko
 7    1 0xffffffff81012000 1e2c     ng_socket.ko
 8    7 0xffffffff81014000 8d38     netgraph.ko
 9    1 0xffffffff8101d000 186a     ng_mppc.ko
10    1 0xffffffff8101f000 28c      rc4.ko
11    1 0xffffffff81020000 2a08     ng_l2tp.ko
12    1 0xffffffff81023000 2080     ng_ksocket.ko
13    1 0xffffffff81026000 aa8      ng_tee.ko
14    1 0xffffffff81027000 13b0     ng_iface.ko
15    1 0xffffffff81029000 45e8     ng_ppp.ko
[root@server6 ~]# kldload ipdivert
[root@server6 ~]# kldstat
Id Refs Address            Size     Name
 1   36 0xffffffff80100000 e56d68   kernel
 2    3 0xffffffff80f57000 1a3e0    ipfw.ko
 3    1 0xffffffff80f72000 192b0    aacu64.ko
 4    1 0xffffffff80f8c000 c470     if_lagg.ko
 5    1 0xffffffff80f99000 4088     ipfw_nat.ko
 6    2 0xffffffff80f9e000 14320    libalias.ko
 7    1 0xffffffff81012000 1e2c     ng_socket.ko
 8    7 0xffffffff81014000 8d38     netgraph.ko
 9    1 0xffffffff8101d000 186a     ng_mppc.ko
10    1 0xffffffff8101f000 28c      rc4.ko
11    1 0xffffffff81020000 2a08     ng_l2tp.ko
12    1 0xffffffff81023000 2080     ng_ksocket.ko
13    1 0xffffffff81026000 aa8      ng_tee.ko
14    1 0xffffffff81027000 13b0     ng_iface.ko
15    1 0xffffffff81029000 45e8     ng_ppp.ko
16    1 0xffffffff8102e000 1649     ipdivert.ko
[root@server6 ~]# kldunload ipdivert
kldunload: can't unload file: Operation not permitted
[root@server6 ~]# kldstat
Id Refs Address            Size     Name
[hangs]
<CTRL+T>
load: 0.00  cmd: kldstat 15366 [kernel linker] 98.04r 0.00u 0.00s 0% 1052k


Logging into the maschine again again.
[root@server6 ~]# ps ax | grep 1536[6]
15366   0  T+     0:00.00 kldstat
[root@server6 ~]# procstat -k 15366
  PID    TID COMM             TDNAME           KSTACK
[hangs]

Even "procstat"ing the hanging procstat hangs.

>Fix:
Reboot

>Release-Note:
>Audit-Trail:

From: Lutz Donnerhacke <lutz@donnerhacke.de>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/174236: "kldunload ipdivert" corrupts kernel
Date: Fri, 7 Dec 2012 15:45:00 +0100

 Any module which returns an error on unload will cause the problem.
 It's not specific to ipdivert.ko or ipfw.ko or anything else.
 
 Relevant code section is:
 
 static int
 div_modevent(module_t mod, int type, void *unused)
 {
         int err = 0;
         switch (type) {
 	[...]
         case MOD_QUIESCE:
 		err = EPERM;
 		break;
 	[...]
 	}
         return err;
 }
 
 Please note: Forced unload of ipdivert.ko works fine.
     

From: Eitan Adler <lists@eitanadler.com>
To: Lutz Donnerhacke <lutz@donnerhacke.de>
Cc: freebsd-bugs@freebsd.org, bug-followup <bug-followup@freebsd.org>
Subject: Re: kern/174236: "kldunload ipdivert" corrupts kernel
Date: Fri, 7 Dec 2012 13:42:29 -0500

 On 7 December 2012 09:50, Lutz Donnerhacke <lutz@donnerhacke.de> wrote:
 > The following reply was made to PR kern/174236; it has been noted by GNATS.
 >
 > From: Lutz Donnerhacke <lutz@donnerhacke.de>
 > To: bug-followup@FreeBSD.org
 > Cc:
 > Subject: Re: kern/174236: "kldunload ipdivert" corrupts kernel
 > Date: Fri, 7 Dec 2012 15:45:00 +0100
 >
 >  Any module which returns an error on unload will cause the problem.
 >  It's not specific to ipdivert.ko or ipfw.ko or anything else.
 >
 >  Relevant code section is:
 >
 >  static int
 >  div_modevent(module_t mod, int type, void *unused)
 >  {
 >          int err = 0;
 >          switch (type) {
 >         [...]
 >          case MOD_QUIESCE:
 >                 err = EPERM;
 >                 break;
 >         [...]
 >         }
 >          return err;
 >  }
 >
 >  Please note: Forced unload of ipdivert.ko works fine.
 
 If you are able and willing to debug:
 
 <avg> I'd recommend him to enter debugger and run ps there
 <avg> and get backtraces of the hanging threads
 <dwhite-> it sounds like a leaked lock in the error case
 
 If not, just leaving this here for the record ;)
 
 
 -- 
 Eitan Adler
>Unformatted:
