From nobody@FreeBSD.org  Mon Oct 15 18:14:29 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 717ED16A420
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 15 Oct 2007 18:14:29 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 5C4B913C468
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 15 Oct 2007 18:14:29 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.1/8.14.1) with ESMTP id l9FIELq3009198
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 15 Oct 2007 18:14:21 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.1/8.14.1/Submit) id l9FIELQT009197;
	Mon, 15 Oct 2007 18:14:21 GMT
	(envelope-from nobody)
Message-Id: <200710151814.l9FIELQT009197@www.freebsd.org>
Date: Mon, 15 Oct 2007 18:14:21 GMT
From: Anders Nordby <anders@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: FreeBSD 7-PRERELEASE crashes upon load when running Varnish trunk
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         117216
>Category:       kern
>Synopsis:       [ipfilter] FreeBSD 7-PRERELEASE crashes upon load when running Varnish trunk
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    darrenr
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Oct 15 18:20:00 UTC 2007
>Closed-Date:    Tue Jan 22 14:57:49 UTC 2008
>Last-Modified:  Tue Jan 22 14:57:49 UTC 2008
>Originator:     Anders Nordby
>Release:        7-PRERELASE, amd64
>Organization:
Aftenposten AS
>Environment:
FreeBSD cache12.xxxx.no 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #4: Mon Oct 15 13:13:13 CEST 2007     root@cache12.xxxx.no:/usr/obj/usr/src/sys/CACHE12  amd64
>Description:
After giving Varnish some load, FreeBSD kernel crashes:

login: Sleeping thread (tid 100038, pid 31) owns a non-sleepable lock
sched_switch() at sched_switch+0x184
mi_switch() at mi_switch+0x189
sleepq_wait() at sleepq_wait+0x3b
_sx_slock_hard() at _sx_slock_hard+0x19d
fr_check() at fr_check+0x2b7
pfil_run_hooks() at pfil_run_hooks+0x9c
ip_output() at ip_output+0x339
tcp_output() at tcp_output+0x982
tcp_do_segment() at tcp_do_segment+0x9f8
tcp_input() at tcp_input+0x759
ip_input() at ip_input+0xa8
ether_demux() at ether_demux+0x1b4
ether_input() at ether_input+0x1bb
bce_intr() at bce_intr+0x24f
ithread_loop() at ithread_loop+0x180
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffae3f9d30, rbp = 0 ---
panic: sleeping thread
cpuid = 6
KDB: enter: panic
[thread pid 1170 tid 100523 ]
Stopped at      kdb_enter+0x31: leave
db> bt
Tracing pid 1170 tid 100523 td 0xffffff00229869c0
kdb_enter() at kdb_enter+0x31
panic() at panic+0x173
propagate_priority() at propagate_priority+0x1ec
turnstile_wait() at turnstile_wait+0x1be
_mtx_lock_sleep() at _mtx_lock_sleep+0x9e
in_getsockaddr() at in_getsockaddr+0xb3
kern_getsockname() at kern_getsockname+0x71
getsockname() at getsockname+0x63
syscall() at syscall+0x254
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (32, FreeBSD ELF64, getsockname), rip = 0x800c640ec, rsp = 0x7fffdb2d85e8, rbp = 0x7fffdb2d86c0 ---
db> 

I have two quad-core processors like this:

CPU: Intel(R) Xeon(R) CPU           X5355  @ 2.66GHz (2666.78-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f7  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
  AMD Features=0x20000800<SYSCALL,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4

>How-To-Repeat:
1) Install FreeBSD RELENG_7. Mine is as of october 15.

2) Install Varnish/trunk (up to date to commit 2096), from http://varnish.projects.linpro.no/.

3) Start Varnish. Preferrably on a SMP system with several data files for storage. I use an 8-core system with 8 GB RAM and 3 data files on separate RAID volumes.

4) Give Varnish load.

>Fix:
N/A


>Release-Note:
>Audit-Trail:

From: Anders Nordby <anders@FreeBSD.org>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc:  
Subject: Re: kern/117216: FreeBSD 7-PRERELEASE crashes upon load when running Varnish trunk
Date: Sat, 20 Oct 2007 19:24:39 +0200

 I should note that I was running IP Filter on this system. Removing IP
 Filter, the problem goes away.
 
 This is related to PR 117182?
 
 -- 
 Anders.
Responsible-Changed-From-To: freebsd-bugs->darrenr 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Oct 22 01:55:44 UTC 2007 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=117216 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/117216: commit references a PR
Date: Tue, 30 Oct 2007 22:40:17 +0000 (UTC)

 darrenr     2007-10-30 15:23:27 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/contrib/ipfilter/netinet fil.c ip_auth.c ip_compat.h 
                                  ip_fil_freebsd.c ip_log.c 
                                  ip_nat.c ip_state.c 
   Log:
   Apply a few changes from ipfilter-current:
   * Do not hold any locks over calls to copyin/copyout.
   * Clean up some #ifdefs
   * fix a possible mbuf leak when NAT fails on policy routed packets
   
   PR:             117216
   
   Revision  Changes    Path
   1.54      +4 -4      src/sys/contrib/ipfilter/netinet/fil.c
   1.46      +1 -1      src/sys/contrib/ipfilter/netinet/ip_auth.c
   1.35      +1 -1      src/sys/contrib/ipfilter/netinet/ip_compat.h
   1.8       +7 -6      src/sys/contrib/ipfilter/netinet/ip_fil_freebsd.c
   1.35      +6 -5      src/sys/contrib/ipfilter/netinet/ip_log.c
   1.44      +44 -26    src/sys/contrib/ipfilter/netinet/ip_nat.c
   1.41      +6 -1      src/sys/contrib/ipfilter/netinet/ip_state.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/117216: commit references a PR
Date: Wed, 31 Oct 2007 09:01:58 +0000 (UTC)

 darrenr     2007-10-31 05:00:38 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_7)
     contrib/ipfilter     HISTORY Makefile ip_fil.c md5.h radix.c 
                          radix_ipf.h 
     contrib/ipfilter/BSD Makefile kupgrade 
     contrib/ipfilter/iplang Makefile 
     contrib/ipfilter/ipsend iptests.c sock.c 
     contrib/ipfilter/l4check Makefile l4check.c 
     contrib/ipfilter/lib Makefile alist_new.c ipft_tx.c printnat.c 
                          printpacket.c printpool_live.c 
                          printstate.c 
     contrib/ipfilter/man ippool.5 
     contrib/ipfilter/test Makefile dotest nattest test.format 
     contrib/ipfilter/test/expected f11 i21 in1 in6 
     contrib/ipfilter/test/input f11 l1 
     contrib/ipfilter/test/regress i21 i3 in1 in6 
     contrib/ipfilter/tools ipf_y.y ipfstat.c ipmon.c ipnat.c 
                            ipnat_y.y lexer.c 
     sys/contrib/ipfilter/netinet fil.c ip_auth.c ip_compat.h 
                                  ip_fil.h ip_fil_freebsd.c 
                                  ip_frag.c ip_htable.c ip_log.c 
                                  ip_lookup.c ip_lookup.h ip_nat.c 
                                  ip_nat.h ip_pool.c ip_pool.h 
                                  ip_proxy.c ip_rpcb_pxy.c 
                                  ip_scan.c ip_state.c ip_state.h 
                                  ip_sync.c ipl.h mlfk_ipl.c 
   Log:
   MFC the following:
   Apply a few changes from ipfilter-current:
   * Do not hold any locks over calls to copyin/copyout.
   * Clean up some #ifdefs
   * fix a possible mbuf leak when NAT fails on policy routed packets
   
   PR:             117216
   Approved by:    re
   
   Revision      Changes    Path
   1.1.1.12.2.1  +10 -2     src/contrib/ipfilter/BSD/Makefile
   1.1.1.7.2.1   +9 -5      src/contrib/ipfilter/BSD/kupgrade
   1.1.1.27.2.1  +99 -1     src/contrib/ipfilter/HISTORY
   1.7.2.1       +9 -12     src/contrib/ipfilter/Makefile
   1.5.2.1       +7 -4      src/contrib/ipfilter/ip_fil.c
   1.1.1.4.10.1  +5 -4      src/contrib/ipfilter/iplang/Makefile
   1.13.2.1      +7 -3      src/contrib/ipfilter/ipsend/iptests.c
   1.18.2.1      +7 -3      src/contrib/ipfilter/ipsend/sock.c
   1.1.1.1.24.1  +1 -1      src/contrib/ipfilter/l4check/Makefile
   1.2.10.1      +31 -14    src/contrib/ipfilter/l4check/l4check.c
   1.1.1.4.2.1   +1 -7      src/contrib/ipfilter/lib/Makefile
   1.1.1.1.2.1   +5 -3      src/contrib/ipfilter/lib/alist_new.c
   1.6.2.1       +24 -13    src/contrib/ipfilter/lib/ipft_tx.c
   1.4.2.1       +10 -5     src/contrib/ipfilter/lib/printnat.c
   1.4.2.1       +3 -3      src/contrib/ipfilter/lib/printpacket.c
   1.1.1.1.2.1   +9 -4      src/contrib/ipfilter/lib/printpool_live.c
   1.5.2.1       +3 -3      src/contrib/ipfilter/lib/printstate.c
   1.2.10.1      +2 -2      src/contrib/ipfilter/man/ippool.5
   1.2.10.1      +2 -2      src/contrib/ipfilter/md5.h
   1.4.2.1       +7 -1      src/contrib/ipfilter/radix.c
   1.4.2.1       +3 -3      src/contrib/ipfilter/radix_ipf.h
   1.1.1.16.2.1  +14 -10    src/contrib/ipfilter/test/Makefile
   1.1.1.4.2.1   +7 -1      src/contrib/ipfilter/test/dotest
   1.1.1.2.24.1  +124 -0    src/contrib/ipfilter/test/expected/f11
   1.1.1.1.2.1   +6 -0      src/contrib/ipfilter/test/expected/i21
   1.1.1.5.2.1   +1 -0      src/contrib/ipfilter/test/expected/in1
   1.1.1.2.2.1   +1 -0      src/contrib/ipfilter/test/expected/in6
   1.1.1.3.10.1  +11 -11    src/contrib/ipfilter/test/input/f11
   1.1.1.2.10.1  +8 -8      src/contrib/ipfilter/test/input/l1
   1.1.1.2.10.1  +8 -1      src/contrib/ipfilter/test/nattest
   1.1.1.1.2.1   +1 -0      src/contrib/ipfilter/test/regress/i21
   1.1.1.3.10.1  +4 -2      src/contrib/ipfilter/test/regress/i3
   1.1.1.4.2.1   +1 -0      src/contrib/ipfilter/test/regress/in1
   1.1.1.2.2.1   +1 -0      src/contrib/ipfilter/test/regress/in6
   1.1.1.4.2.1   +4 -1      src/contrib/ipfilter/test/test.format
   1.6.2.1       +25 -1     src/contrib/ipfilter/tools/ipf_y.y
   1.6.2.1       +4 -4      src/contrib/ipfilter/tools/ipfstat.c
   1.7.2.1       +33 -4     src/contrib/ipfilter/tools/ipmon.c
   1.5.2.1       +63 -4     src/contrib/ipfilter/tools/ipnat.c
   1.5.2.1       +2 -1      src/contrib/ipfilter/tools/ipnat_y.y
   1.4.2.1       +40 -17    src/contrib/ipfilter/tools/lexer.c
   1.52.2.1      +164 -125  src/sys/contrib/ipfilter/netinet/fil.c
   1.44.2.1      +19 -19    src/sys/contrib/ipfilter/netinet/ip_auth.c
   1.33.2.1      +127 -57   src/sys/contrib/ipfilter/netinet/ip_compat.h
   1.35.2.1      +32 -21    src/sys/contrib/ipfilter/netinet/ip_fil.h
   1.6.2.1       +136 -149  src/sys/contrib/ipfilter/netinet/ip_fil_freebsd.c
   1.32.2.1      +9 -9      src/sys/contrib/ipfilter/netinet/ip_frag.c
   1.4.2.1       +40 -52    src/sys/contrib/ipfilter/netinet/ip_htable.c
   1.33.2.1      +22 -16    src/sys/contrib/ipfilter/netinet/ip_log.c
   1.1.1.3.2.1   +43 -7     src/sys/contrib/ipfilter/netinet/ip_lookup.c
   1.1.1.3.2.1   +1 -2      src/sys/contrib/ipfilter/netinet/ip_lookup.h
   1.42.2.1      +189 -69   src/sys/contrib/ipfilter/netinet/ip_nat.c
   1.26.2.1      +7 -3      src/sys/contrib/ipfilter/netinet/ip_nat.h
   1.1.1.3.2.1   +36 -49    src/sys/contrib/ipfilter/netinet/ip_pool.c
   1.1.1.3.2.1   +2 -2      src/sys/contrib/ipfilter/netinet/ip_pool.h
   1.29.2.1      +7 -5      src/sys/contrib/ipfilter/netinet/ip_proxy.c
   1.1.1.3.2.1   +1 -1      src/sys/contrib/ipfilter/netinet/ip_rpcb_pxy.c
   1.1.1.4.2.1   +4 -2      src/sys/contrib/ipfilter/netinet/ip_scan.c
   1.39.2.1      +109 -65   src/sys/contrib/ipfilter/netinet/ip_state.c
   1.19.2.1      +5 -7      src/sys/contrib/ipfilter/netinet/ip_state.h
   1.5.2.1       +6 -6      src/sys/contrib/ipfilter/netinet/ip_sync.c
   1.26.2.1      +5 -5      src/sys/contrib/ipfilter/netinet/ipl.h
   1.19.2.1      +11 -2     src/sys/contrib/ipfilter/netinet/mlfk_ipl.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->feedback 
State-Changed-By: linimon 
State-Changed-When: Mon Nov 5 06:29:05 UTC 2007 
State-Changed-Why:  
To submitter: did this commit fix your problem? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=117216 

From: Anders Nordby <anders@FreeBSD.org>
To: linimon@FreeBSD.org
Cc: darrenr@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: kern/117216: [ipfilter] FreeBSD 7-PRERELEASE crashes upon load when running Varnish trunk
Date: Sun, 30 Dec 2007 20:33:06 +0100

 Hi,
 
 While I unfortunately have not been able to try this with 7-current yet,
 I do see a crash that happens rather often in 6.3-PRERELEASE (up to date
 to 30 december) which has the same version of IP Filter (4.1.28):
 
 Fatal trap 12: page fault while in kernel mode
 fault virtual address   = 0xc
 fault code              = supervisor read, page not present
 instruction pointer     = 0x20:0xc05100e7
 stack pointer           = 0x28:0xc7775b28
 frame pointer           = 0x28:0xc7775b4c
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 11 (swi1: net)
 trap number             = 12
 panic: page fault
 KDB: stack backtrace:
 kdb_backtrace(100,c110f780,28,c7775ae8,c,...) at kdb_backtrace+0x29
 panic(c06397a8,c06565e6,0,fffff,c110ea9b,...) at panic+0xa8
 trap_fatal(c7775ae8,c,c110f780,0,c,...) at trap_fatal+0x2a6
 trap_pfault(c7775ae8,0,c) at trap_pfault+0x1f3
 trap(8,28,180028,c11e8d54,588,...) at trap+0x325
 calltrap() at calltrap+0x5
 --- trap 0xc, eip = 0xc05100e7, esp = 0xc7775b28, ebp = 0xc7775b4c ---
 m_copym(0,5dc,5c8,1,14,...) at m_copym+0x2f
 ip_fragment(c134f80e,c7775c04,5dc,0,1,...) at ip_fragmestray irq7
 nt+0x214
 ip_output(c130d800,0,c7775bd0,1,0,0) at ip_output+0x85e
 ip_forward(c130d800,0) at ip_forward+0x280
 ip_input(c130d800) at ip_input+0x59f
 netisr_processqueue(c0698118) at netisr_processqueue+0x9f
 swi_net(0) at swi_net+0xf2
 ithread_execute_handlers(c110ea78,c1101500) at
 ithread_execute_handlers+0x121
 ithread_loop(c10f8770,c7775d38) at ithread_loop+0x54
 fork_exit(c04c3344,c10f8770,c7775d38) at fork_exit+0x70
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0x1, eip = 0, esp = 0xc7775d6c, ebp = 0 ---
 Uptime: 1h12m56s
 Cannot dump. No dump device defined.
 Automatic reboot in 15 seconds - press a key on the console to abort
 Rebooting...
 PC Engines WRAP.1C/1D/1E v1.08
 640 KB Base Memory
 153603174448128645128089697280113664130048 KB Extended Memory
 
 This is on my home firewall:
 
 - Even with just pass in all/pass out all rules.
 
 - Nat rules:
 
 map ath0 192.168.78.0/24 -> 0/32 proxy port ftp ftp/tcp
 map ath0 192.168.78.0/24 -> 0/32 proxy port 500 ipsec/udp
 map ath0 192.168.78.0/24 -> 0/32 portmap tcp/udp 40000:60000
 map ath0 192.168.78.0/24 -> 0/32
 
 - Typically happens when I rsync large datasets through it...
 
 This might be a different bug than this PR originally was about. I'll
 try to get that checked soonish.
 
 On Mon, Nov 05, 2007 at 06:29:49AM +0000, linimon@FreeBSD.org wrote:
 > Synopsis: [ipfilter] FreeBSD 7-PRERELEASE crashes upon load when running Varnish trunk
 > 
 > State-Changed-From-To: open->feedback
 > State-Changed-By: linimon
 > State-Changed-When: Mon Nov 5 06:29:05 UTC 2007
 > State-Changed-Why: 
 > To submitter: did this commit fix your problem?
 > 
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=117216
 
 
 -- 
 Anders.

From: Anders Nordby <anders@fupp.net>
To: linimon@FreeBSD.org
Cc: darrenr@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: kern/117216: [ipfilter] FreeBSD 7-PRERELEASE crashes upon load when running Varnish trunk
Date: Mon, 31 Dec 2007 12:03:53 +0100

 Hi,
 
 On Sun, Dec 30, 2007 at 08:33:06PM +0100, Anders Nordby wrote:
 > panic: page fault
 > KDB: stack backtrace:
 > kdb_backtrace(100,c110f780,28,c7775ae8,c,...) at kdb_backtrace+0x29
 > panic(c06397a8,c06565e6,0,fffff,c110ea9b,...) at panic+0xa8
 > trap_fatal(c7775ae8,c,c110f780,0,c,...) at trap_fatal+0x2a6
 > trap_pfault(c7775ae8,0,c) at trap_pfault+0x1f3
 > trap(8,28,180028,c11e8d54,588,...) at trap+0x325
 > calltrap() at calltrap+0x5
 > --- trap 0xc, eip = 0xc05100e7, esp = 0xc7775b28, ebp = 0xc7775b4c ---
 > m_copym(0,5dc,5c8,1,14,...) at m_copym+0x2f
 > ip_fragment(c134f80e,c7775c04,5dc,0,1,...) at ip_fragmestray irq7
 > nt+0x214
 > ip_output(c130d800,0,c7775bd0,1,0,0) at ip_output+0x85e
 > ip_forward(c130d800,0) at ip_forward+0x280
 > ip_input(c130d800) at ip_input+0x59f
 > netisr_processqueue(c0698118) at netisr_processqueue+0x9f
 > swi_net(0) at swi_net+0xf2
 > ithread_execute_handlers(c110ea78,c1101500) at
 > ithread_execute_handlers+0x121
 > ithread_loop(c10f8770,c7775d38) at ithread_loop+0x54
 > fork_exit(c04c3344,c10f8770,c7775d38) at fork_exit+0x70
 > fork_trampoline() at fork_trampoline+0x8
 > --- trap 0x1, eip = 0, esp = 0xc7775d6c, ebp = 0 ---
 > Uptime: 1h12m56s
 > Cannot dump. No dump device defined.
 > Automatic reboot in 15 seconds - press a key on the console to abort
 > Rebooting...
 > PC Engines WRAP.1C/1D/1E v1.08
 > 640 KB Base Memory
 > 153603174448128645128089697280113664130048 KB Extended Memory
 
 I'm sorry, but this also happens with PF. The problem seems to be with
 sis interfaces and polling. After turning off polling on my sis
 interface, I don't get these panics anymore.
 
 As said, I'll get back to the original problem for this PR.
 
 Bye,
 
 -- 
 Anders.
State-Changed-From-To: feedback->closed 
State-Changed-By: darrenr 
State-Changed-When: Tue Jan 22 14:55:40 UTC 2008 
State-Changed-Why:  
This bug was raised against ipfilter and some potential fixes offered. 
The submitter now believes it is an sis driver problem, so i'd like to 
close this and encourage the original submitter to file a new bug. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=117216 
>Unformatted:
Follow up ping. Will close if no feedback is received.
