From netch@segfault.kiev.ua  Wed Jul 15 20:51:31 2009
Return-Path: <netch@segfault.kiev.ua>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 13BD31065729
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 15 Jul 2009 20:51:31 +0000 (UTC)
	(envelope-from netch@segfault.kiev.ua)
Received: from segfault.kiev.ua (segfault.kiev.ua [193.193.193.4])
	by mx1.freebsd.org (Postfix) with ESMTP id AF8568FC15
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 15 Jul 2009 20:51:29 +0000 (UTC)
	(envelope-from netch@segfault.kiev.ua)
Received: from segfault.kiev.ua (localhost.segfault.kiev.ua [127.0.0.1])
	by segfault.kiev.ua (8.14.3/8.14.3/8.Who.Cares) with ESMTP id n6FKdnM0003518;
	Wed, 15 Jul 2009 23:39:49 +0300 (EEST)
	(envelope-from netch@segfault.kiev.ua)
Received: (from netch@localhost)
	by segfault.kiev.ua (8.14.3/8.14.3/Submit) id n6FKdne2003515;
	Wed, 15 Jul 2009 23:39:49 +0300 (EEST)
	(envelope-from netch)
Message-Id: <200907152039.n6FKdne2003515@segfault.kiev.ua>
Date: Wed, 15 Jul 2009 23:39:49 +0300 (EEST)
From: Valentin Nechayev <netch@segfault.kiev.ua>
Reply-To: Valentin Nechayev <netch@segfault.kiev.ua>
To: FreeBSD-gnats-submit@freebsd.org
Subject: Kernel panic and hanging on using SCTP
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         136803
>Category:       kern
>Synopsis:       [sctp] [panic] Kernel panic and hanging on using SCTP
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-net
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jul 15 21:00:05 UTC 2009
>Closed-Date:    Sun Aug 09 09:49:26 UTC 2009
>Last-Modified:  Sun Aug 09 09:49:26 UTC 2009
>Originator:     Valentin Nechayev
>Release:        FreeBSD 7.2-RELEASE i386
>Organization:
private
>Environment:

FreeBSD 7.2-RELEASE / i386
FreeBSD 7.2-RELEASE-p1 / i386

>Description:

A small test was written to expose some standard SCTP using. When running,
approx. at 3-5-th client connect kernel is crashing to panic or hanging
without any reaction to keyboard or network. Programs weren't started under
root:) and couldn't give any side effect to system.

I have got kernel crash vmcore, kgdb shows:

=== cut ===
Unread portion of the kernel message buffer:
panic: hashdestroy: hash not empty
Uptime: 1d13h5m9s
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc0536730 in boot (howto=260) at /usr/BSD/src/sys/kern/kern_shutdown.c:418
#2  0xc0536931 in panic (fmt=Variable "fmt" is not available.
) at /usr/BSD/src/sys/kern/kern_shutdown.c:574
#3  0xc053d211 in hashdestroy (vhashtbl=0xc5a29400, type=0xc07d46c0, 
    hashmask=31) at /usr/BSD/src/sys/kern/kern_subr.c:415
#4  0xc0638383 in sctp_inpcb_free (inp=0xc3d43cc0, immediate=0, from=1)
    at /usr/BSD/src/sys/netinet/sctp_pcb.c:3419
#5  0xc0643e24 in sctp_close (so=0xc58e0340)
    at /usr/BSD/src/sys/netinet/sctp_usrreq.c:623
#6  0xc0589be3 in soclose (so=0xc58e0340)
    at /usr/BSD/src/sys/kern/uipc_socket.c:667
#7  0xc057121b in soo_close (fp=0xc44ce2f8, td=0xc59aed20)
    at /usr/BSD/src/sys/kern/sys_socket.c:273
#8  0xc05015d3 in fdrop (fp=0xc44ce2f8, td=0xc59aed20) at file.h:300
#9  0xc0502b7f in closef (fp=0xc44ce2f8, td=0xc59aed20)
    at /usr/BSD/src/sys/kern/kern_descrip.c:2036
#10 0xc0503d75 in fdfree (td=0xc59aed20)
    at /usr/BSD/src/sys/kern/kern_descrip.c:1745
#11 0xc0511378 in exit1 (td=0xc59aed20, rv=256)
    at /usr/BSD/src/sys/kern/kern_exit.c:284
#12 0xc051271d in sys_exit (td=Could not find the frame base for "sys_exit".
) at /usr/BSD/src/sys/kern/kern_exit.c:110
#13 0xc0759685 in syscall (frame=0xe37bbd38)
    at /usr/BSD/src/sys/i386/i386/trap.c:1090
#14 0xc0746c30 in Xint0x80_syscall ()
    at /usr/BSD/src/sys/i386/i386/exception.s:255
#15 0x00000033 in ?? ()
=== end cut ===

>How-To-Repeat:

Code for test server and test client is attached in unformatted part.
Start server in one terminal, then run client a few times in another terminal.

>Fix:

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Jul 17 04:15:38 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=136803 

From: Bruce Cran <bruce@cran.org.uk>
To: bug-followup@FreeBSD.org, netch@segfault.kiev.ua
Cc:  
Subject: Re: kern/136803: [sctp] [panic] Kernel panic and hanging on using
 SCTP
Date: Fri, 7 Aug 2009 23:41:13 +0100

 It looks like revision 1.62.2.17 of /sys/netinet/sctp_pcb.c fixed this
 crash. Could you confirm whether the problem has been fixed?
 
 -- 
 Bruce

From: Valentin Nechayev <netch@netch.kiev.ua>
To: Bruce Cran <bruce@cran.org.uk>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/136803: [sctp] [panic] Kernel panic and hanging on using
	SCTP
Date: Sat, 8 Aug 2009 10:06:06 +0300

 Hi,
 
  Fri, Aug 07, 2009 at 23:41:13, bruce wrote about "Re: kern/136803: [sctp] [panic] Kernel panic and hanging on using SCTP": 
 
 > It looks like revision 1.62.2.17 of /sys/netinet/sctp_pcb.c fixed this
 > crash. Could you confirm whether the problem has been fixed?
 
 With this version, can't reproduce anymore.
 Thanks!
 
 
 -netch-
State-Changed-From-To: open->closed 
State-Changed-By: bms 
State-Changed-When: Sun 9 Aug 2009 09:49:09 UTC 
State-Changed-Why:  
I committed the fix. Oops 

http://www.freebsd.org/cgi/query-pr.cgi?pr=136803 
>Unformatted:

Server code:

=== cut ===
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/sctp.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <netdb.h>
#include <string.h>
#include <stdio.h>
#include <err.h>

static int
server(int s, struct sockaddr* sa, socklen_t sl)
{   
    char host[200];
    char service[100];
    int gni = getnameinfo(sa, sl, host, sizeof(host),
            service, sizeof(service), NI_NUMERICHOST|NI_NUMERICSERV);
    if (gni == 0) {
        printf("Connect from %s:%s\n", host, service);
    }
    else {
        printf("Address error: %s\n", gai_strerror(gni));
        close(s);
        return;
    }
    struct sctp_sndrcvinfo sinfo;
    // Send greeting
    memset(&sinfo, 0, sizeof(sinfo));
    sinfo.sinfo_stream = 1234;
    if (sctp_send(s, "hi", 2, &sinfo, 0) < 0)
        err(1, "sctp_send");
    // XXX
    close(s);
}

int
main()
{
    struct sockaddr_in sia;
    int ss = -1;

    ss = socket(AF_INET, SOCK_STREAM, IPPROTO_SCTP);
    if (ss < 0)
        err(1, "socket(SCTP)");
    memset(&sia, 0, sizeof(sia));
    sia.sin_family = AF_INET;
    sia.sin_addr.s_addr = htonl(0x7F000001);
    sia.sin_port = htons(5210);
    if (bind(ss, (struct sockaddr*)&sia, sizeof(sia)) < 0)
        err(1, "bind()");
    if (listen(ss, 1) < 0)
        err(1, "listen()");
    printf("Started to listen, ss=%d\n", ss);
    for(;;) {
        socklen_t sl;
        int sconn;
        sl = sizeof(sia);
        sconn = accept(ss, (struct sockaddr*)&sia, &sl);
        if (sconn < 0) {
            warn("accept()");
            usleep(20000);
            continue;
        }
        server(sconn, (struct sockaddr*)&sia, sl);
    }
    // UNREACHED
    return 0;
}
=== end cut ===

Client code:

=== cut ===
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/sctp.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <netdb.h>
#include <string.h>
#include <stdio.h>
#include <err.h>

int
main()
{
    struct sockaddr_in sia;
    int ss = -1;
    struct sctp_sndrcvinfo sinfo;
    int rflags;
    char buf[200];

    ss = socket(AF_INET, SOCK_STREAM, IPPROTO_SCTP);
    if (ss < 0)
        err(1, "socket(SCTP)");
    memset(&sia, 0, sizeof(sia));
    sia.sin_family = AF_INET;
    sia.sin_addr.s_addr = htonl(0x7F000001);
    sia.sin_port = htons(5210);
    if (connect(ss, (struct sockaddr*)&sia, sizeof(sia)) < 0)
        err(1, "connect()");
    printf("Connected\n");
    if (sctp_recvmsg(ss, buf, sizeof(buf), NULL, 0, &sinfo, &rflags) < 0)
        err(1, "recvmsg()");
    printf("Got data for stream %u\n", (unsigned) sinfo.sinfo_stream);
    close(ss);
    return 0;
}
=== end cut ===

Kernel config (for machine where hanged):

=== cut ===
cpu             I686_CPU
ident           nn72

# To statically compile in device wiring instead of /boot/device.hints
#hints          "GENERIC.hints"         # Default places to look for devices.

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols

options         SCHED_ULE               # ULE scheduler
options         PREEMPTION              # Enable kernel thread preemption
options         INET                    # InterNETworking
options         INET6
options         SCTP                    # Stream Control Transmission Protocol
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         UFS_ACL                 # Support for access control lists
options         UFS_DIRHASH             # Improve performance on big directories
options         UFS_GJOURNAL            # Enable gjournal-based UFS journaling
options         NFSCLIENT               # Network Filesystem Client
options         NFSSERVER               # Network Filesystem Server
options         NFSLOCKD                # Network Lock Manager
options         MSDOSFS                 # MSDOS Filesystem
options         CD9660                  # ISO 9660 Filesystem
options         PROCFS                  # Process filesystem (requires PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         GEOM_PART_GPT           # GUID Partition Tables.
options         GEOM_LABEL              # Provides labelization
options         COMPAT_43TTY            # BSD 4.3 TTY compat [KEEP THIS!]
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         COMPAT_FREEBSD5         # Compatible with FreeBSD5
options         COMPAT_FREEBSD6         # Compatible with FreeBSD6
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI
options         KTRACE                  # ktrace(1) support
options         STACK                   # stack(9) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options         ADAPTIVE_GIANT          # Giant mutex is adaptive.
options         STOP_NMI                # Stop CPUS using NMI instead of IPI
options         AUDIT                   # Security event auditing
options         KDTRACE_HOOKS           # Kernel DTrace hooks

# To make an SMP kernel, the next two lines are needed
options         SMP                     # Symmetric MultiProcessor Kernel
device          apic                    # I/O APIC

# CPU frequency control
device          cpufreq

# Bus support.
device          eisa
device          pci

# Floppy drives
device          fdc

# ATA and ATAPI devices
device          ata
device          atadisk         # ATA disk drives
device          ataraid         # ATA RAID drives
device          atapicd         # ATAPI CDROM drives
options         ATA_STATIC_ID   # Static device numbering

# SCSI peripherals
device          scbus           # SCSI bus (required for SCSI)
device          da              # Direct Access (disks)
device          cd              # CD
device          pass            # Passthrough device (direct SCSI access)
device          atapicam

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard
device          psm             # PS/2 mouse

device          kbdmux          # keyboard multiplexer

device          vga             # VGA video card driver

device          splash          # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device          sc

device          agp             # support several AGP chipsets

# Power management support (see NOTES for more options)
#device         apm
# Add suspend/resume support for the i8254.
device          pmtimer

# Serial (COM) ports
device          sio             # 8250, 16[45]50 based serial ports
device          uart            # Generic UART driver

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device          miibus          # MII bus support

# Pseudo devices.
device          loop            # Network loopback
device          random          # Entropy device
device          ether           # Ethernet support
device          pty             # Pseudo-ttys (telnet etc)
device          md              # Memory "disks"
device          gif             # IPv6 and IPv4 tunneling
device          firmware        # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device          bpf             # Berkeley packet filter

# Mandatory:
device          apic                    # I/O apic

device          speaker         #Play IBM BASIC-style noises out your speaker

options         INCLUDE_CONFIG_FILE     # Include this file in kernel
options         MSGBUF_SIZE=131072

options         NETGRAPH                # netgraph(4) system
options         IPFIREWALL              #firewall
options         IPFIREWALL_VERBOSE      #enable logging to syslogd(8)
options         IPFIREWALL_VERBOSE_LIMIT=100    #limit verbosity
options         IPFIREWALL_DEFAULT_TO_ACCEPT    #allow everything by default
options         IPFIREWALL_FORWARD      #packet destination changes
options         IPDIVERT                #divert sockets
options         DUMMYNET

options         SC_HISTORY_SIZE=1200    # number of history buffer lines
options         SC_MOUSE_CHAR=0x3       # char code for text mode mouse cursor
=== end cut ===
