From jfh@cise.ufl.edu  Wed Aug 27 04:18:32 2003
Return-Path: <jfh@cise.ufl.edu>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 5716116A4BF
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 27 Aug 2003 04:18:32 -0700 (PDT)
Received: from mail.cise.ufl.edu (fir.cise.ufl.edu [128.227.205.249])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 9A5B743FEC
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 27 Aug 2003 04:18:31 -0700 (PDT)
	(envelope-from jfh@cise.ufl.edu)
Received: from palm.cise.ufl.edu (palm.cise.ufl.edu [128.227.205.231])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.cise.ufl.edu (Postfix) with ESMTP
	id DCA6C375; Wed, 27 Aug 2003 07:18:30 -0400 (EDT)
Received: (from jfh@localhost)
	by palm.cise.ufl.edu (8.12.9/8.12.6/Submit) id h7RBIU0T041697;
	Wed, 27 Aug 2003 07:18:30 -0400 (EDT)
Message-Id: <200308271118.h7RBIU0T041697@palm.cise.ufl.edu>
Date: Wed, 27 Aug 2003 07:18:30 -0400 (EDT)
From: "James F. Hranicky" <jfh@cise.ufl.edu>
Reply-To: "James F. Hranicky" <jfh@cise.ufl.edu>
To: FreeBSD-gnats-submit@freebsd.org
Cc: admin@cise.ufl.edu
Subject: ipfw hangs on every invocation
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         56031
>Category:       kern
>Synopsis:       [ipfw] ipfw hangs on every invocation
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    luigi
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Aug 27 04:20:14 PDT 2003
>Closed-Date:    
>Last-Modified:  Fri Oct 28 06:43:25 GMT 2005
>Originator:     James F. Hranicky
>Release:        FreeBSD 4.8-STABLE i386
>Organization:
University of Florida CISE Department
>Environment:
System: FreeBSD palm.cise.ufl.edu 4.8-STABLE FreeBSD 4.8-STABLE #30: Tue Aug 12 11:51:33 EDT 2003 root@palm.cise.ufl.edu:/private/freebsd-src/obj/private/freebsd-src/src/sys/CISEKERN i386

>Description:
	Upon upgrade to 4.8-STABLE #30: Tue Aug 12 11:51:33 EDT 2003, ipfw hangs each
        time it is called. 

        All my machines upgraded to the version above have ipfw's left hung by
        /etc/periodic/security/500.ipfwdenied. 

        Those still at 4.8-STABLE #28: Tue Jun 10 09:21:14 do not seem to have
        this problem. 

        I do not currently use ipfw on any machine for firewalling capabilities, it
        is compiled into my kernel in case I need it. I have the default set to allow
        all packets through.
>How-To-Repeat:
	Unknown.
>Fix:
	Unknown


>Release-Note:
>Audit-Trail:

From: Robert Watson <rwatson@FreeBSD.org>
To: "James F. Hranicky" <jfh@cise.ufl.edu>
Cc: FreeBSD-gnats-submit@FreeBSD.org, admin@cise.ufl.edu,
	freebsd-bugs@FreeBSD.org
Subject: Re: conf/56031: ipfw hangs on every invocation
Date: Thu, 28 Aug 2003 10:07:55 -0400 (EDT)

 On Wed, 27 Aug 2003, James F. Hranicky wrote:
 
 > 	Upon upgrade to 4.8-STABLE #30: Tue Aug 12 11:51:33 EDT 2003, ipfw hangs each
 >         time it is called. 
 > 
 >         All my machines upgraded to the version above have ipfw's left hung by
 >         /etc/periodic/security/500.ipfwdenied. 
 > 
 >         Those still at 4.8-STABLE #28: Tue Jun 10 09:21:14 do not seem to have
 >         this problem. 
 > 
 >         I do not currently use ipfw on any machine for firewalling capabilities, it
 >         is compiled into my kernel in case I need it. I have the default set to allow
 >         all packets through.
 
 Could you send the output of "ps axlwww" showing the hung ipfw process, as
 well as its parent process?  When you run ipfw on the command line using
 the same arguments found in the periodic script, does it hang?  If so,
 what do you see when you press Ctrl-T?
 
 Thanks,
 
 Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
 robert@fledge.watson.org      Network Associates Laboratories
 
 

From: "James F.  Hranicky" <jfh@cise.ufl.edu>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: FreeBSD-gnats-submit@FreeBSD.org, admin@cise.ufl.edu,
	freebsd-bugs@FreeBSD.org
Subject: Re: conf/56031: ipfw hangs on every invocation
Date: Tue, 2 Sep 2003 12:02:42 -0400

 On Thu, 28 Aug 2003 10:07:55 -0400 (EDT)
 Robert Watson <rwatson@FreeBSD.org> wrote:
 
 > 
 > On Wed, 27 Aug 2003, James F. Hranicky wrote:
 > 
 > > 	Upon upgrade to 4.8-STABLE #30: Tue Aug 12 11:51:33 EDT 2003, ipfw hangs each
 > >         time it is called. 
 
 [ ... ] 
 
 > Could you send the output of "ps axlwww" showing the hung ipfw process, as
 > well as its parent process?  When you run ipfw on the command line using
 > the same arguments found in the periodic script, does it hang?  If so,
 > what do you see when you press Ctrl-T?
 
 Yes, any invocation of ipfw hangs. Here is the strace output -- oddly enough, 
 it looks like some kind of terminal issue:
 
     # strace -f ipfw list
     execve("/sbin/ipfw", ["ipfw", "list"], [/* 27 vars */]) = 0
     readlink("/etc/malloc.conf", 0xbfbff950, 63) = -1 ENOENT (No such file or directory)
     mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0x28087000
     break(0x8098000)                        = 0
     break(0x8099000)                        = 0
     ioctl(0, TIOCGETA, {B38400 opost isig icanon echo ...}) = 0
     break(0x809a000)                        = 0
     socket(PF_INET, SOCK_RAW, IPPROTO_RAW)  = 3
     getsockopt(3, IPPROTO_IP, 54, [0], [176]) = 0
     fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(5, 0), ...}) = 0
     break(0x809b000)                        = 0
     ioctl(1, TIOCGETA, {B38400 opost isig icanon echo ...}) = 0
 
 The parent process is /bin/sh when called from cron:
 
     # p | grep ipfw
     root   root   43032 42979  0.0  0.1   644  260  ??  I    Sat03AM   0:00.00 
     /bin/sh - /etc/periodic/security/500.ipfwdenied
 
     root   root   43034 43032 23.7  0.0   488  164  ??  R    Sat03AM 2459:26.28 ipfw -a l
 
     root   root   56542 56489  0.0  0.1   644  264  ??  I    Sun03AM   0:00.00 
     /bin/sh - /etc/periodic/security/500.ipfwdenied
 
     root   root   56544 56542 23.9  0.0   488  164  ??  R    Sun03AM 1164:21.71 ipfw -a l
 
     root   root   67717 67664  0.0  0.1   644  264  ??  I    Mon03AM   0:00.00 
     /bin/sh - /etc/periodic/security/500.ipfwdenied
 
     root   root   67719 67717 24.2  0.0   488  164  ??  R    Mon03AM 512:16.35 ipfw -a l
 
     root   root   78835 78782  0.0  0.1   644  268  ??  I     3:01AM   0:00.00 
     /bin/sh - /etc/periodic/security/500.ipfwdenied
 
     root   root   78837 78835 23.5  0.0   488  164  ??  R     3:01AM  83:18.34 ipfw -a l
 
 The parent process is zsh from the command line:
 
     0   377   357   0  18  0  2248 1964 pause  I     p0    0:00.11 zsh
     0   449   377 139  45  0   464  288 -      R+    p0    0:32.93 ipfw list
 
 Ctrl-T shows this when run from the command line:
     
     <root@palm:~> # ipfw list
     load: 0.32  cmd: ipfw 396 [running] 1.45u 0.00s 4% 288k
     load: 0.32  cmd: ipfw 396 [running] 3.04u 0.00s 9% 288k
     load: 0.32  cmd: ipfw 396 [running] 4.68u 0.00s 18% 288k
     load: 0.37  cmd: ipfw 396 [running] 6.09u 0.00s 25% 288k
     load: 0.37  cmd: ipfw 396 [running] 7.79u 0.00s 29% 288k
     load: 0.37  cmd: ipfw 396 [running] 9.47u 0.00s 36% 288k
     load: 0.42  cmd: ipfw 396 [running] 11.12u 0.00s 42% 288k
     
 This goes to 99% CPU utiliztion quickly.
 
 An upgrade to 
 
      4.9-PRERELEASE #31: Tue Sep  2 08:35:32
 
 does not seem to fix the problem.
 
 Let me know if I can provide any more info.
 
 Jim

From: Robert Watson <rwatson@FreeBSD.org>
To: "James F.  Hranicky" <jfh@cise.ufl.edu>
Cc: FreeBSD-gnats-submit@FreeBSD.org, admin@cise.ufl.edu,
	freebsd-bugs@FreeBSD.org
Subject: Re: conf/56031: ipfw hangs on every invocation
Date: Tue, 2 Sep 2003 12:10:28 -0400 (EDT)

 On Tue, 2 Sep 2003, James F.  Hranicky wrote:
 
 > Let me know if I can provide any more info. 
 
 This seems to suggest it's really just the userland application spinning,
 not a hang in the kernel.  Could you try using truss to see if, once it
 starts spinning, it's making system calls, or just stuck entirely in
 userspace?
   
 Assuming it's purely a userspace problem (perhaps triggered by
 syntactically poor output from the kernel), the next thing to do is
 probably to instrument your ipfw binary with either printfs or debugging
 symbols and see where in its execution it is spinning.  Could you include
 a list of your IPFW rules also, please?  I wonder if the kernel or
 userland tool contains a bug that is causing it to misbehave for some
 specific ruleset.
 
 Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
 robert@fledge.watson.org      Network Associates Laboratories
 
 

From: "James F.  Hranicky" <jfh@cise.ufl.edu>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: FreeBSD-gnats-submit@FreeBSD.org, admin@cise.ufl.edu,
	freebsd-bugs@FreeBSD.org
Subject: Re: conf/56031: ipfw hangs on every invocation
Date: Wed, 3 Sep 2003 06:28:09 -0400

 On Tue, 2 Sep 2003 12:10:28 -0400 (EDT)
 Robert Watson <rwatson@FreeBSD.org> wrote:
 
 > 
 > On Tue, 2 Sep 2003, James F.  Hranicky wrote:
 > 
 > > Let me know if I can provide any more info. 
 > 
 > This seems to suggest it's really just the userland application spinning,
 > not a hang in the kernel.  Could you try using truss to see if, once it
 > starts spinning, it's making system calls, or just stuck entirely in
 > userspace?
 
 As far as I can tell it's hanging in the last ioctl() .
 
 > Assuming it's purely a userspace problem (perhaps triggered by
 > syntactically poor output from the kernel), the next thing to do is
 > probably to instrument your ipfw binary with either printfs or debugging
 > symbols and see where in its execution it is spinning.  
 
 I can compile it with debugging symbols and trace through the execution, 
 but it seems like the ioctl() is where it's hanging, again some kind of
 odd terminal thing.
 
 > Could you include a list of your IPFW rules also, please?  
 
 I don't have any set at this time, just the default pass any any. 
 
 I'll trace through it with gdb if you think it will help.
 
 Jim

From: Robert Watson <rwatson@FreeBSD.org>
To: "James F.  Hranicky" <jfh@cise.ufl.edu>
Cc: FreeBSD-gnats-submit@FreeBSD.org, admin@cise.ufl.edu,
	freebsd-bugs@FreeBSD.org
Subject: Re: conf/56031: ipfw hangs on every invocation
Date: Wed, 3 Sep 2003 11:45:41 -0400 (EDT)

 On Wed, 3 Sep 2003, James F.  Hranicky wrote:
 
 > On Tue, 2 Sep 2003 12:10:28 -0400 (EDT)
 > Robert Watson <rwatson@FreeBSD.org> wrote:
 > 
 > > 
 > > On Tue, 2 Sep 2003, James F.  Hranicky wrote:
 > > 
 > > > Let me know if I can provide any more info. 
 > > 
 > > This seems to suggest it's really just the userland application spinning,
 > > not a hang in the kernel.  Could you try using truss to see if, once it
 > > starts spinning, it's making system calls, or just stuck entirely in
 > > userspace?
 > 
 > As far as I can tell it's hanging in the last ioctl() .
 
 In, or after?  The "Running" state reported by Ctrl-T suggests it is in
 userspace spinning, perhaps as a result of an unexpected return from the
 ioctl()...
 
 > > Assuming it's purely a userspace problem (perhaps triggered by
 > > syntactically poor output from the kernel), the next thing to do is
 > > probably to instrument your ipfw binary with either printfs or debugging
 > > symbols and see where in its execution it is spinning.  
 > 
 > I can compile it with debugging symbols and trace through the execution, 
 > but it seems like the ioctl() is where it's hanging, again some kind of
 > odd terminal thing.
 > 
 > > Could you include a list of your IPFW rules also, please?  
 > 
 > I don't have any set at this time, just the default pass any any. 
 > 
 > I'll trace through it with gdb if you think it will help.
 
 I'd step up to the ioctl in question, and then see if it really hangs in
 the ioctl(), or if it gets past and starts spinning.  If it's the ioctl(),
 it would be very helpful to know which file descriptor it's on, and what
 the arguments are.  If it's not the ioctl() call, we need to figure out
 which loop isn't taking something important into account.
 
 Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
 robert@fledge.watson.org      Network Associates Laboratories
 
 

From: "James F.  Hranicky" <jfh@cise.ufl.edu>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: FreeBSD-gnats-submit@FreeBSD.org, admin@cise.ufl.edu,
	freebsd-bugs@FreeBSD.org
Subject: Re: conf/56031: ipfw hangs on every invocation
Date: Thu, 4 Sep 2003 08:01:42 -0400

 On Wed, 3 Sep 2003 11:45:41 -0400 (EDT)
 Robert Watson <rwatson@FreeBSD.org> wrote:
 
 > On Wed, 3 Sep 2003, James F.  Hranicky wrote:
 
 > I'd step up to the ioctl in question, and then see if it really hangs in
 > the ioctl(), or if it gets past and starts spinning.  If it's the ioctl(),
 > it would be very helpful to know which file descriptor it's on, and what
 > the arguments are.  If it's not the ioctl() call, we need to figure out
 > which loop isn't taking something important into account.
 
 I think I've found it. First off, I neglected to mention I've added
 
 	IPFW2=TRUE
 
 to /etc/make.conf, so I'm using ipfw2 (sorry!).
 
 After tracing through, it does seem to be a logic error in ipfw2.c .
 
 At line 1036, there a loop like so:
 
         for (l = rule->act_ofs, cmd = rule->cmd ;
                         l > 0 ; l -= F_LEN(cmd) , cmd += F_LEN(cmd)) {
 
 The loop is supposed to quit when l == 0, and l is supposed to be decremented
 by F_LEN(cmd) each time.
 
 F_LEN is defined as
 
 	#define   F_LEN_MASK      0x3f
 	#define   F_LEN(cmd)      ((cmd)->len & F_LEN_MASK)
 
 Checking the values of the cmd struct:
 
 	(gdb) p *cmd
 	$6 = {opcode = O_NOP, len = 0 '\000', arg1 = 0}
 
 Darn, l is always decremented by 0.
 
 I'll probably fix the problem on my end by temporarily disabling ipfw2 (since I'm
 not really even using it at the moment). 
 
 Hope this helps.
 
 Jim
Responsible-Changed-From-To: freebsd-bugs->luigi 
Responsible-Changed-By: luigi 
Responsible-Changed-When: Sat Sep 6 07:38:43 PDT 2003 
Responsible-Changed-Why:  
will look at it, thanks for the very detailed analysis of the bug 

luigi 


http://www.freebsd.org/cgi/query-pr.cgi?pr=56031 

From: Stephen Gill <gillsr@cymru.com>
To: <bug-followup@FreeBSD.org>, <jfh@cise.ufl.edu>
Cc:  
Subject: Re: kern/56031: ipfw hangs on every invocation
Date: Tue, 03 May 2005 22:40:21 -0500

 Any updates on this PR?  I'm seeing the same issue with IPFW2 on FreeBSD
 4.11-RELEASE.
 
 Cheers,
 -- steve
 
 
>Unformatted:
