From nobody@FreeBSD.org  Thu Mar 11 14:19:53 2004
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 1B4A016A4CE
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 11 Mar 2004 14:19:53 -0800 (PST)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 0106243D2D
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 11 Mar 2004 14:19:53 -0800 (PST)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.12.10/8.12.10) with ESMTP id i2BMJq72029058
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 11 Mar 2004 14:19:52 -0800 (PST)
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.12.10/8.12.10/Submit) id i2BMJqCi029057;
	Thu, 11 Mar 2004 14:19:52 -0800 (PST)
	(envelope-from nobody)
Message-Id: <200403112219.i2BMJqCi029057@www.freebsd.org>
Date: Thu, 11 Mar 2004 14:19:52 -0800 (PST)
From: Gerrit Nagelhout <gnagelhout@sandvine.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [patch] kernel panic using kqueue with EVFILT_TIMER and timeout 0
X-Send-Pr-Version: www-2.3

>Number:         64121
>Category:       kern
>Synopsis:       [patch] kernel panic using kqueue with EVFILT_TIMER and timeout 0
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    cperciva
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Mar 11 14:20:17 PST 2004
>Closed-Date:    Tue Apr 06 23:01:19 PDT 2004
>Last-Modified:  Tue Apr 06 23:01:19 PDT 2004
>Originator:     Gerrit Nagelhout
>Release:        4.7
>Organization:
Sandvine
>Environment:
# uname -a
FreeBSD TPC-D2-36 4.74.0002. FreeBSD 4.74.0002. #1: 20040309_21:15:07   i386 4.74.0002
This is our own build of FREEBSD 4.7 with some merges from 4.9, and some local modifications.
      
>Description:
This panic almost always occured in an SMP system running at 2500hz.

#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:492
492     /usr/src/sys/kern/kern_shutdown.c: No such file or directory.
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:492
#1  0xc01bef28 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:321
#2  0xc01bf451 in panic (fmt=0xc033e439 "%s")
    at /usr/src/sys/kern/kern_shutdown.c:607
#3  0xc02e5ade in trap_fatal (frame=0xe03bad90, eva=20)
    at /usr/src/sys/i386/i386/trap.c:992
#4  0xc02e570d in trap_pfault (frame=0xe03bad90, usermode=0, eva=20)
    at /usr/src/sys/i386/i386/trap.c:885
#5  0xc02e51af in trap (frame={tf_fs = -533004264, tf_es = -678035440, 
      tf_ds = 720912, tf_edi = 1073741824, tf_esi = 1, tf_ebp = -532959780, 
      tf_isp = -532959812, tf_ebx = 0, tf_edx = 1, tf_ecx = 0, 
      tf_eax = 1073741824, tf_trapno = 12, tf_err = 0, tf_eip = -1071882413, 
      tf_cs = 8, tf_eflags = 66067, tf_esp = -531521856, tf_ss = -1071957392})
    at /usr/src/sys/i386/i386/trap.c:484
#6  0xc01c5f53 in callout_reset (c=0x0, to_ticks=1, 
    ftn=0xc01b3a70 <filt_timerexpire>, arg=0xe0519ec0)
    at /usr/src/sys/kern/kern_timeout.c:244
#7  0xc01b3aeb in filt_timerexpire (knx=0xe0519ec0)
    at /usr/src/sys/kern/kern_event.c:294
#8  0xc01c5e4e in softclock () at /usr/src/sys/kern/kern_timeout.c:140
#9  0xc02d46cb in doreti_swi ()
#10 0xc01b3b7b in filt_timerattach (kn=0xe0519ec0)
    at /usr/src/sys/kern/kern_event.c:320
---Type <return> to continue, or q <return> to quit---
#11 0xc01b4034 in kqueue_register (kq=0xc35fd500, kev=0xc35fd570, p=0xdc34f8e0)
    at /usr/src/sys/kern/kern_event.c:529
#12 0xc01b3d74 in kevent (p=0xdc34f8e0, uap=0xe03baf80)
    at /usr/src/sys/kern/kern_event.c:414
#13 0xc02e5e2d in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, 
      tf_edi = -1077936956, tf_esi = -1077936948, tf_ebp = -1077937040, 
      tf_isp = -532959276, tf_ebx = 1, tf_edx = 4096, tf_ecx = 3, 
      tf_eax = 363, tf_trapno = 12, tf_err = 2, tf_eip = 1745495508, 
      tf_cs = 31, tf_eflags = 663, tf_esp = -1078019004, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1193
#14 0xc02d061b in Xint0x80_syscall ()
#15 0x8048472 in ?? ()


The problem is that in filt_timerattach, the kn_hook is set after callout_reset.  With a timeout of zero (and high hz) it is possible that the callout occurs before callout_reset returns.  In this case, the value of kn_hook is invalid.  This pointer will then be used by the callout, and if the ONESHOT flag is not set, it will be used to add as the next callout, causing a panic on some pointer values, and general corruption on others.
      
>How-To-Repeat:

Run:


#include <sys/types.h>
#include <sys/event.h>

int main()
{
    int i;
    struct kevent ev[4096];
    int kq = kqueue();
    for (i = 0; i < 4096; ++i)
    {   
        EV_SET(ev + i, i, EVFILT_TIMER, EV_ADD, 0, 0, 0);
    }
    kevent(kq, ev, 4096, 0, 0, 0);
    return 0;
}      
>Fix:
*** kern_event.c.orig
--- kern_event.c
***************
*** 331,338 ****
        MALLOC(calloutp, struct callout *, sizeof(*calloutp),
            M_KQUEUE, M_WAITOK);
        callout_init(calloutp);
-       callout_reset(calloutp, tticks, filt_timerexpire, kn);
        kn->kn_hook = (caddr_t)calloutp;
  
        return (0);
  }
--- 331,338 ----
        MALLOC(calloutp, struct callout *, sizeof(*calloutp),
            M_KQUEUE, M_WAITOK);
        callout_init(calloutp);
        kn->kn_hook = (caddr_t)calloutp;
+       callout_reset(calloutp, tticks, filt_timerexpire, kn);
  
        return (0);
  }
      
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->cperciva 
Responsible-Changed-By: cperciva 
Responsible-Changed-When: Wed Mar 17 21:23:03 PST 2004 
Responsible-Changed-Why:  
I'm working on this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64121 
State-Changed-From-To: open->closed 
State-Changed-By: cperciva 
State-Changed-When: Tue Apr 6 23:01:02 PDT 2004 
State-Changed-Why:  
Fixed, thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64121 
>Unformatted:
