From kostik@kib.kiev.ua  Thu May 22 10:27:01 2003
Return-Path: <kostik@kib.kiev.ua>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id A0B0D37B401
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 22 May 2003 10:27:01 -0700 (PDT)
Received: from smtp3.lucky.net (smtp3.lucky.net [62.244.55.204])
	by mx1.FreeBSD.org (Postfix) with ESMTP id D75C743F75
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 22 May 2003 10:26:58 -0700 (PDT)
	(envelope-from kostik@kib.kiev.ua)
Received: from smtp.lucky.net (smtp1.lucky.net [193.193.193.117])
	by smtp3.lucky.net (postfix-1.1.5) with ESMTP id 24BD08990A
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 22 May 2003 20:26:56 +0300 (EEST)
Received: from kozlik.carrier.kiev.ua (smmsp@kozlik.carrier.kiev.ua [193.193.193.111])
	by smtp.lucky.net  with ESMTP id h5MHQp2D069430
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 22 May 2003 20:26:54 +0300 (EEST)
	(envelope-from kostik@kib.kiev.ua)
Received: (from uucp@localhost)
	by kozlik.carrier.kiev.ua  with UUCP id h5MHQi04034555
	for FreeBSD-gnats-submit@freebsd.org; Thu, 22 May 2003 20:26:44 +0300 (EEST)
	(envelope-from kostik@kib.kiev.ua)
Received: from kib.UUCP (uucp@localhost)
	by kozlik.carrier.kiev.ua (rmail mypid=34554 childpid=34555) with UUCP;
	Thu, 22 May 2003 17:26:44 +0000 GMT
Received: from little.home (kostik@localhost [127.0.0.1])
	by little.home (8.12.8p1/8.12.8) with ESMTP id h4MHNPTR000432
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 22 May 2003 20:23:25 +0300 (EEST)
	(envelope-from kostik@little.home)
Received: (from kostik@localhost)
	by little.home (8.12.8p1/8.12.8/Submit) id h4MHNPiP000431;
	Thu, 22 May 2003 20:23:25 +0300 (EEST)
	(envelope-from kostik)
Message-Id: <200305221723.h4MHNPiP000431@little.home>
Date: Thu, 22 May 2003 20:23:25 +0300 (EEST)
From: "Kostik I. Belousov" <kostik@kib.kiev.ua>
Reply-To: "Kostik I. Belousov" <kostya@tessart.kiev.ua>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Kernel panic with ipfw2 and syncookies
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         52585
>Category:       kern
>Synopsis:       [netinet] [patch] Kernel panic with ipfw2 and syncookies
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-net
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu May 22 10:30:03 PDT 2003
>Closed-Date:    Sun Nov 12 09:40:16 GMT 2006
>Last-Modified:  Sun Nov 12 09:40:16 GMT 2006
>Originator:     Kostik I. Belousov
>Release:        FreeBSD 4.8-RELEASE i386
>Organization:
none
>Environment:
System: FreeBSD little.home 4.8-RELEASE FreeBSD 4.8-RELEASE #1: Fri May 2 18:08:25 EEST 2003 root@little.home:/usr/obj/usr/src/sys/LITTLE i386

sysctl
hw.machine: i386
hw.model: Pentium II/Pentium II Xeon/Celeron
hw.ncpu: 2
hw.byteorder: 1234
hw.physmem: 533917696
hw.usermem: 491724800
hw.pagesize: 4096
hw.floatingpoint: 1
hw.machine_arch: i386
hw.ata.ata_dma: 1
hw.ata.wc: 1
hw.ata.tags: 0
hw.ata.atapi_dma: 0
hw.instruction_sse: 0
hw.availpages: 130185

net.inet.tcp.syncookies: 1
net.inet.tcp.syncache.bucketlimit: 30
net.inet.tcp.syncache.cachelimit: 15359
net.inet.tcp.syncache.count: 0
net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.rexmtlimit: 3

Kernel compiled with ipfw2.

>Description:
	By adding/removing aliases and manipulating ipfw rules
(I caused the panic using ipfw fwd, see below), the kernel could
be paniced. The trace:
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc0158847 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc0158cb9 in panic (fmt=0xc0291b19 "%s")
    at /usr/src/sys/kern/kern_shutdown.c:595
#3  0xc024b459 in trap_fatal (frame=0xd3933cc0, eva=8)
    at /usr/src/sys/i386/i386/trap.c:974
#4  0xc024b0c5 in trap_pfault (frame=0xd3933cc0, usermode=0, eva=8)
    at /usr/src/sys/i386/i386/trap.c:867
#5  0xc024ac1f in trap (frame={tf_fs = 1644167192, tf_es = -1072234480,
      tf_ds = -745340912, tf_edi = 1644167168, tf_esi = -1054094552,
      tf_ebp = -745325300, tf_isp = -745325332, tf_ebx = -761024704,
      tf_edx = -1070824920, tf_ecx = 0, tf_eax = -1, tf_trapno = 12,
      tf_err = 0, tf_eip = -1071929334, tf_cs = 8, tf_eflags = 66198,
      tf_esp = -761024704, tf_ss = -1050054796})
    at /usr/src/sys/i386/i386/trap.c:466
#6  0xc01ba80a in syncache_insert (sc=0xd2a3af40, sch=0xc12bcb28)
    at /usr/src/sys/netinet/tcp_syncache.c:302
#7  0xc01bb67c in syncache_add (inc=0xd3933db4, to=0xd3933e20, th=0xc1051950,
    sop=0xd3933db0, m=0xc1051900) at /usr/src/sys/netinet/tcp_syncache.c:1021
#8  0xc01b5809 in tcp_input (m=0xc1051900, off0=20, proto=6)
    at /usr/src/sys/netinet/tcp_input.c:826
#9  0xc01b026c in ip_input (m=0xc1051900)
    at /usr/src/sys/netinet/ip_input.c:927
#10 0xc01b02cb in ipintr () at /usr/src/sys/netinet/ip_input.c:948
#11 0xc023c051 in swi_net_next ()
#12 0xc017a835 in connect (p=0xd3876be0, uap=0xd3933f80)
    at /usr/src/sys/kern/uipc_syscalls.c:394
#13 0xc024b795 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
      tf_edi = 135174508, tf_esi = -1077953148, tf_ebp = -1077953136,
      tf_isp = -745324588, tf_ebx = 0, tf_edx = 134570369, tf_ecx = 135112576,
      tf_eax = 98, tf_trapno = 22, tf_err = 2, tf_eip = 673579312, tf_cs = 31,
      tf_eflags = 659, tf_esp = -1077953564, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1175
#14 0xc0237f5b in Xint0x80_syscall ()
#15 0x805b4b5 in ?? ()
#16 0x8059c0e in ?? ()
#17 0x805985f in ?? ()
#18 0x806e639 in ?? ()
#19 0x804c03a in ?? ()
(kgdb) frame 6
#6  0xc01ba80a in syncache_insert (sc=0xd2a3af40, sch=0xc12bcb28)
    at /usr/src/sys/netinet/tcp_syncache.c:302
302                             if (sc2 != NULL)
(kgdb) list
297                      * first non-empty timer queue with the largest
298                      * timeout value.
299                      */
300                     for (i = SYNCACHE_MAXREXMTS; i >= 0; i--) {
301                             sc2 = TAILQ_FIRST(&tcp_syncache.timerq[i]);
302                             if (sc2 != NULL)
303                                     break;
304                     }
305                     sc2->sc_tp->ts_recent = ticks;
306                     syncache_drop(sc2, NULL);

>How-To-Repeat:
	ifconfig lo0 192.168.2.1 alias
	ipfw 50 add fwd 192.168.2.1,23 tcp from any to 192.168.2.1
some time ...
	ipfw del 50
	ifconfig lo0 192.168.2.1 remove
some more time ...
attempt to make tcp connection to the machine panics the kernel


>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->luigi 
Responsible-Changed-By: kris 
Responsible-Changed-When: Mon Jul 14 03:07:48 PDT 2003 
Responsible-Changed-Why:  
Assign to ipfw maintainer 

http://www.freebsd.org/cgi/query-pr.cgi?pr=52585 

From: Don Bowman <don@sandvine.com>
To: "'freebsd-gnats-submit@FreeBSD.org'" <freebsd-gnats-submit@FreeBSD.org>,
	"'tessart.kiev.ua'" <freebsd-gnats-submit@FreeBSD.org, kostya>
Cc:  
Subject: Re: kern/52585: Kernel panic with ipfw2 and syncookies
Date: Tue, 29 Jul 2003 20:12:00 -0400

 I believe i have a fix for this. The problem
 is not in ipfw per se, but in syncache which doesn't 
 expect to 're-enter' itself. when ipfw is in use,
 ip_output() may call ip_input().
 
 http://www.mail-archive.com/freebsd-net@freebsd.org/msg08996.html
 has the trail.
 
 the below fix may not be great since it defers deletion.
 It also may be better to run the deferred deletion @ the beginning
 of the timer routine.
 
 Index: tcp_syncache.c
 ===================================================================
 RCS file: /usr/cvs/src/sys/netinet/tcp_syncache.c,v
 retrieving revision 1.5.2.8.1000.3
 diff -U5 -r1.5.2.8.1000.3 tcp_syncache.c
 --- tcp_syncache.c      4 Feb 2003 01:52:03 -0000       1.5.2.8.1000.3
 +++ tcp_syncache.c      1 Jul 2003 14:32:29 -0000
 @@ -83,16 +83,18 @@
  #endif /*IPSEC*/
  
  #include <machine/in_cksum.h>
  #include <vm/vm_zone.h>
  
 +static int syncache_delete_flag;
  static int tcp_syncookies = 1;
  SYSCTL_INT(_net_inet_tcp, OID_AUTO, syncookies, CTLFLAG_RW,
      &tcp_syncookies, 0, 
      "Use TCP SYN cookies if the syncache overflows");
  
  static void     syncache_drop(struct syncache *, struct syncache_head *);
 +static void     syncache_delete(struct syncache *, struct syncache_head *);
  static void     syncache_free(struct syncache *);
  static void     syncache_insert(struct syncache *, struct syncache_head *);
  struct syncache *syncache_lookup(struct in_conninfo *, struct syncache_head
 **);
  static int      syncache_respond(struct syncache *, struct mbuf *);
  static struct   socket *syncache_socket(struct syncache *, struct socket
 *);
 @@ -125,10 +127,11 @@
         u_int   next_reseed;
         TAILQ_HEAD(, syncache) timerq[SYNCACHE_MAXREXMTS + 1];
         struct  callout tt_timerq[SYNCACHE_MAXREXMTS + 1];
  };
  static struct tcp_syncache tcp_syncache;
 +static TAILQ_HEAD(syncache_delete_list, syncache)      sc_delete_list;
  
  SYSCTL_NODE(_net_inet_tcp, OID_AUTO, syncache, CTLFLAG_RW, 0, "TCP SYN
 cache");
  
  SYSCTL_INT(_net_inet_tcp_syncache, OID_AUTO, bucketlimit, CTLFLAG_RD,
       &tcp_syncache.bucket_limit, 0, "Per-bucket hash limit for syncache");
 @@ -202,10 +205,13 @@
                         rtrequest(RTM_DELETE, rt_key(rt),
                             rt->rt_gateway, rt_mask(rt),
                             rt->rt_flags, NULL);
                 RTFREE(rt);
         }
 +#if defined(DIAGNOSTIC)
 +       memset(sc, 0xee, sizeof(struct syncache));
 +#endif
         zfree(tcp_syncache.zone, sc);
  }
  
  void
  syncache_init(void)
 @@ -256,10 +262,12 @@
          * older one.
          */
         tcp_syncache.cache_limit -= 1;
         tcp_syncache.zone = zinit("syncache", sizeof(struct syncache),
             tcp_syncache.cache_limit, ZONE_INTERRUPT, 0);
 +
 +       TAILQ_INIT(&sc_delete_list);
  }
  
  static void
  syncache_insert(sc, sch)
         struct syncache *sc;
 @@ -312,12 +320,28 @@
  static void
  syncache_drop(sc, sch)
         struct syncache *sc;
         struct syncache_head *sch;
  {
 +       if ((sc->sc_flags & SCF_DELETE) == 0) {
 +               sc->sc_flags |= SCF_DELETE;
 +               syncache_delete_flag = 1;
 +               TAILQ_INSERT_TAIL(&sc_delete_list, sc, sc_delete);
 +       }
 +}
 +
 +static void
 +syncache_delete(sc, sch)
 +       struct syncache *sc;
 +       struct syncache_head *sch;
 +{
         int s;
  
 +       if ((sc->sc_flags & SCF_DELETE) == 0) {
 +           printf("ERROR ERROR ERROR: SCF_DELETE == 0\n");
 +           return;
 +       }
         if (sch == NULL) {
  #ifdef INET6
                 if (sc->sc_inc.inc_isipv6) {
                         sch = &tcp_syncache.hashbase[
                             SYNCACHE_HASH6(&sc->sc_inc,
 tcp_syncache.hashmask)];
 @@ -329,10 +353,12 @@
                 }
         }
  
         s = splnet();
  
 +       TAILQ_REMOVE(&sc_delete_list, sc, sc_delete);
 +
         TAILQ_REMOVE(&sch->sch_bucket, sc, sc_hash);
         sch->sch_length--;
         tcp_syncache.cache_count--;
  
         TAILQ_REMOVE(&tcp_syncache.timerq[sc->sc_rxtslot], sc, sc_timerq);
 @@ -357,10 +383,12 @@
         int s;
  
         s = splnet();
          if (callout_pending(&tcp_syncache.tt_timerq[slot]) ||
              !callout_active(&tcp_syncache.tt_timerq[slot])) {
 +               if (syncache_delete_flag)
 +                       goto delete_cleanup;
                  splx(s);
                  return;
          }
          callout_deactivate(&tcp_syncache.tt_timerq[slot]);
  
 @@ -390,10 +418,21 @@
                 SYNCACHE_TIMEOUT(sc, slot + 1);
         }
         if (nsc != NULL)
                 callout_reset(&tcp_syncache.tt_timerq[slot],
                     nsc->sc_rxttime - ticks, syncache_timer, (void
 *)(slot));
 +
 +delete_cleanup:
 +       sc = TAILQ_FIRST(&sc_delete_list);
 +       while (sc != NULL) {
 +               nsc = TAILQ_NEXT(sc, sc_delete);
 +               syncache_delete(sc, NULL); 
 +               sc = nsc;
 +       }
 +       TAILQ_INIT(&sc_delete_list);
 +       syncache_delete_flag = 0;
 +
         splx(s);
  }
  
  /*
   * Find an entry in the syncache.
 @@ -1333,10 +1372,11 @@
         data = data >> SYNCOOKIE_WNDBITS;
  
         sc = zalloc(tcp_syncache.zone);
         if (sc == NULL)
                 return (NULL);
 +       bzero(sc, sizeof(*sc));
         /*
          * Fill in the syncache values.
          * XXX duplicate code from syncache_add
          */
         sc->sc_ipopts = NULL;
 Index: tcp_var.h
 ===================================================================
 RCS file: /usr/cvs/src/sys/netinet/tcp_var.h,v
 retrieving revision 1.56.2.12
 diff -U5 -r1.56.2.12 tcp_var.h
 --- tcp_var.h   24 Aug 2002 18:40:26 -0000      1.56.2.12
 +++ tcp_var.h   1 Jul 2003 02:33:57 -0000
 @@ -222,12 +222,14 @@
  #define SCF_WINSCALE   0x02                    /* negotiated window scaling
 */
  #define SCF_TIMESTAMP  0x04                    /* negotiated timestamps */
  #define SCF_CC         0x08                    /* negotiated CC */
  #define SCF_UNREACH    0x10                    /* icmp unreachable received
 */
  #define SCF_KEEPROUTE  0x20                    /* keep cloned route */
 +#define SCF_DELETE     0x40                    /* I'm being deleted */
         TAILQ_ENTRY(syncache)   sc_hash;
         TAILQ_ENTRY(syncache)   sc_timerq;
 +       TAILQ_ENTRY(syncache)   sc_delete;
  };
  
  struct syncache_head {
         TAILQ_HEAD(, syncache)  sch_bucket;
         u_int           sch_length;
 
Responsible-Changed-From-To: luigi->bms 
Responsible-Changed-By: bms 
Responsible-Changed-When: Wed Jun 23 01:49:31 GMT 2004 
Responsible-Changed-Why:  
I'll try to look at this. 
I have a feeling this problem may go away with the impending 
ipfw/dummynet changes with regards to PFIL_HOOKS. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=52585 

From: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
To: bug-followup@FreeBSD.org, kostya@tessart.kiev.ua
Cc:  
Subject: Re: kern/52585: [netinet] [patch] Kernel panic with ipfw2 and syncookies
Date: Mon, 20 Feb 2006 13:51:50 +0200

 Hello!
 
  I've tried to recreate your panic case under 4.11-RELEASE-p14. I've
 noticed that your ipfw rule
 
 	ipfw 50 add fwd 192.168.2.1,23 tcp from any to 192.168.2.1
 
 creates cyclic redirect of TCP packets directed on this machine
 to 192.168.2.1. E.g., if I enable telnet in /etc/inetd.conf, add this
 rule and "telnet 192.168.2.1 http", rule will not only forward
 packets from telnet'c client to 80th port via 23th port, but will
 also redirect replies (directed in this case to 192.168.2.1:client_port)
 back to 192.168.2.1:23! This provokes system reboot (yes, w/o crash,
 just reset) in several seconds of this telnet session's activity. So
 I've changed this rule to
 
 ipfw 50 add fwd 192.168.2.1,23 tcp from any to 192.168.2.1 http
 
 and then "telnet 192.168.2.1 http" doesn't crash system anymore both under ipfw1 and
 ipfw2. Please reveal what activity should I do during "some time..." and "some more
 time..." in order to provoke panic.
 
 Sincerely, Dmitry
 
 
 
 
    
 
State-Changed-From-To: open->suspended 
State-Changed-By: bms 
State-Changed-When: Fri Sep 22 17:04:22 UTC 2006 
State-Changed-Why:  
Back to the free pool. 
By the looks of things, this problem may have gone away since 5.x, the 
reason being that the network stack was reshuffled for SMP (netisrs 
are now used). 


Responsible-Changed-From-To: bms->freebsd-net 
Responsible-Changed-By: bms 
Responsible-Changed-When: Fri Sep 22 17:04:22 UTC 2006 
Responsible-Changed-Why:  


http://www.freebsd.org/cgi/query-pr.cgi?pr=52585 
State-Changed-From-To: suspended->closed 
State-Changed-By: gnn 
State-Changed-When: Sun Nov 12 09:39:29 UTC 2006 
State-Changed-Why:  
This can no longer be reproduced (7.0 CURRENT) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=52585 
>Unformatted:
