From wawa@sakura-vlan3.yandex.net  Tue Nov  9 16:02:24 2004
Return-Path: <wawa@sakura-vlan3.yandex.net>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 5B27B16A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Tue,  9 Nov 2004 16:02:24 +0000 (GMT)
Received: from sakura-vlan3.yandex.net (sakura-lo0.yandex.net [213.180.192.251])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 42DC943D48
	for <FreeBSD-gnats-submit@freebsd.org>; Tue,  9 Nov 2004 16:02:23 +0000 (GMT)
	(envelope-from wawa@sakura-vlan3.yandex.net)
Received: from sakura-vlan3.yandex.net (localhost [127.0.0.1])
	by sakura-vlan3.yandex.net (8.13.1/8.13.1) with ESMTP id iA9G2LKt088187
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 9 Nov 2004 19:02:21 +0300 (MSK)
	(envelope-from wawa@sakura-vlan3.yandex.net)
Received: (from root@localhost)
	by sakura-vlan3.yandex.net (8.13.1/8.13.1/Submit) id iA9G2L7B088186;
	Tue, 9 Nov 2004 19:02:21 +0300 (MSK)
	(envelope-from wawa)
Message-Id: <200411091602.iA9G2L7B088186@sakura-vlan3.yandex.net>
Date: Tue, 9 Nov 2004 19:02:21 +0300 (MSK)
From: Vladimir Ivanov <wawa@yandex-team.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Page fault in bpf_mtap ()
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         73719
>Category:       kern
>Synopsis:       Page fault in bpf_mtap ()
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    csjp
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Nov 09 16:10:24 GMT 2004
>Closed-Date:    Sun Jan 21 02:59:07 GMT 2007
>Last-Modified:  Sun Jan 21 02:59:07 GMT 2007
>Originator:     Charlie &
>Release:        FreeBSD 5.3-STABLE i386
>Organization:
Yandex LLC
>Environment:
System: FreeBSD sakura.yandex.net 5.3-STABLE FreeBSD 5.3-STABLE #12: Thu Nov 4 22:02:05 MSK 2004 root@sakura.yandex.net:/usr/obj/usr/src/sys/SAKURA i386


	
>Description:

Sometimes FreeBSD router panics in bpf_mtap(). I can note that we are using bfps (with tcpdump e.g.) often on this PC.

sakura:/var/crash# kgdb  -c vmcore.1 /boot/kernel/kernel
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
(no debugging symbols found)...0xc04f3da9 in doadump ()
(kgdb) bt
#0  0xc04f3da9 in doadump ()
#1  0xc04f44a6 in boot ()
#2  0xc04f4854 in panic ()
#3  0xc062a951 in trap_fatal ()
#4  0xc062a648 in trap_pfault ()
#5  0xc062a22d in trap ()
#6  0xc0616c2a in calltrap ()
#7  0xc55f0018 in ?? ()
#8  0xc22e0010 in ?? ()
#9  0xe4a90010 in ?? ()
#10 0xc275b802 in ?? ()
#11 0xc1f85800 in ?? ()
#12 0xe4a98c44 in ?? ()
#13 0xe4a98c10 in ?? ()
#14 0x00000800 in ?? ()
#15 0x00000040 in ?? ()
#16 0x000005ea in ?? ()
#17 0x00000000 in ?? ()
#18 0x0000000c in ?? ()
#19 0x00000000 in ?? ()
#20 0xc0568bcc in bpf_mtap ()
#21 0xc056fafd in ether_input ()
#22 0xc0480cd1 in em_process_receive_interrupts ()
#23 0xc047d9ff in em_intr ()
#24 0xc04dc01f in ithread_loop ()
#25 0xc04dadb3 in fork_exit ()
#26 0xc0616c8c in fork_trampoline ()

>How-To-Repeat:
Frequent opening/closing of bpf device should be enough I think to reproduce the problem.

>Fix:

We suppose that the problem is caused by unlocked modification of if_bpf pointer. 
So, the suspicious place may be fixed by following patch:

--- bpf.h.orig  Sun May 30 21:03:48 2004
+++ bpf.h       Tue Nov  9 18:37:52 2004
@@ -473,9 +473,10 @@
                bpf_tap((_ifp)->if_bpf, (_pkt), (_pktlen));     \
 } while (0)
 #define        BPF_MTAP(_ifp,_m) do {                                  \
-       if ((_ifp)->if_bpf) {                                   \
-               M_ASSERTVALID(_m);                              \
-               bpf_mtap((_ifp)->if_bpf, (_m));                 \
+       struct bpf_if* bbb = (_ifp)->if_bpf;                    \
+       if (bbb) {                                              \
+               M_ASSERTVALID(_m);                              \
+               bpf_mtap(bbb, (_m));                            \
        }                                                       \
 } while (0)
 #define        BPF_MTAP2(_ifp,_data,_dlen,_m) do {                     \



>Release-Note:
>Audit-Trail:

From: Maxim Konovalov <maxim@macomnet.ru>
To: Vladimir Ivanov <wawa@yandex-team.ru>
Cc: bug-followup@freebsd.org
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Tue, 9 Nov 2004 19:16:18 +0300 (MSK)

 debug.mpsafenet=1 per chance?
 
 -- 
 Maxim Konovalov

From: Vladimir Ivanov <wawa@yandex-team.ru>
To: freebsd-gnats-submit@FreeBSD.org, wawa@yandex-team.ru
Cc:  
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Tue, 09 Nov 2004 19:59:50 +0300

 Yes, sure. GIANT has been disabled:
 
 debug.mpsafenet: 1
 

From: Maxim Konovalov <maxim@macomnet.ru>
To: Vladimir Ivanov <wawa@yandex-team.ru>
Cc: bug-followup@freebsd.org
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Tue, 9 Nov 2004 21:02:34 +0300 (MSK)

 On Tue, 9 Nov 2004, 17:00-0000, Vladimir Ivanov wrote:
 
 > The following reply was made to PR kern/73719; it has been noted by GNATS.
 >
 > From: Vladimir Ivanov <wawa@yandex-team.ru>
 > To: freebsd-gnats-submit@FreeBSD.org, wawa@yandex-team.ru
 > Cc:
 > Subject: Re: kern/73719: Page fault in bpf_mtap ()
 > Date: Tue, 09 Nov 2004 19:59:50 +0300
 >
 >  Yes, sure. GIANT has been disabled:
 >
 >  debug.mpsafenet: 1
 
 So could you please try debug.mpsafenet=0?  AFAIK debug.mpsafenet=1 is
 very far from the production state.
 
 -- 
 Maxim Konovalov

From: Vladimir Ivanov <wawa@yandex-team.ru>
To: Maxim Konovalov <maxim@macomnet.ru>
Cc: bug-followup@freebsd.org
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Tue, 09 Nov 2004 21:58:06 +0300

 Maxim Konovalov wrote:
 
 >So could you please try debug.mpsafenet=0?  AFAIK debug.mpsafenet=1 is
 >very far from the production state.
 >
 >  
 >
 I don't think so. I can't say that GIANT affects system stability 
 significantly.
 Anyway, we've found a bug and this bug may be fixed (e.g. with above 
 patch).  Why not ?
 :-)
 

From: Maxim Konovalov <maxim@macomnet.ru>
To: Vladimir Ivanov <wawa@yandex-team.ru>
Cc: bug-followup@freebsd.org
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Tue, 9 Nov 2004 22:15:18 +0300 (MSK)

 On Tue, 9 Nov 2004, 19:00-0000, Vladimir Ivanov wrote:
 
 > The following reply was made to PR kern/73719; it has been noted by GNATS.
 >
 > From: Vladimir Ivanov <wawa@yandex-team.ru>
 > To: Maxim Konovalov <maxim@macomnet.ru>
 > Cc: bug-followup@freebsd.org
 > Subject: Re: kern/73719: Page fault in bpf_mtap ()
 > Date: Tue, 09 Nov 2004 21:58:06 +0300
 >
 >  Maxim Konovalov wrote:
 >
 >  >So could you please try debug.mpsafenet=0?  AFAIK debug.mpsafenet=1 is
 >  >very far from the production state.
 >  >
 >  >
 >  >
 >  I don't think so. I can't say that GIANT affects system stability
 >  significantly.
 >  Anyway, we've found a bug and this bug may be fixed (e.g. with above
 >  patch).  Why not ?
 >  :-)
 
 Because I do not understand what is the root of the bug and what your
 diff fixes, do you?
 
 -- 
 Maxim Konovalov

From: Vladimir Ivanov <wawa@yandex-team.ru>
To: freebsd-gnats-submit@FreeBSD.org, wawa@yandex-team.ru
Cc:  
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Tue, 09 Nov 2004 23:02:26 +0300

 Ok,
 The bpf_mtap () seems to be little enough to make a look.
 
 We suppose that most probable reason to panic is zero value of "bp" pointer.
 Also, I know that bpf open/close are frequent on my system.
 We can see (look at BPF_MTAP definition) that the value may be changed 
 from another thread after verification but before bpf_mtap call because 
 "ifp" points to global variable. The patch does not change the logic of 
 program as you can see but garantee "bp" is not NULL. The only side 
 effect is hypotetic pushing extra packet to just detached bpf device. 
 It's not very big price I seem
 
 I've commited the patch to the system and awaiting results.
 
 

From: Vladimir Ivanov <wawa@yandex-team.ru>
To: Maxim Konovalov <maxim@macomnet.ru>
Cc: bug-followup@freebsd.org
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Tue, 09 Nov 2004 23:12:48 +0300

 >> I don't think so. I can't say that GIANT affects system stability
 >> significantly.
 >> Anyway, we've found a bug and this bug may be fixed (e.g. with above
 >> patch).  Why not ?
 >> :-)
 >>    
 >>
 >
 >Because I do not understand what is the root of the bug and what your
 >diff fixes, do you?
 >
 >  
 >
 Hura!
 I've found my kernel debug info.
 So, we can see full picture ( bp == NULL !).
 
 
 [GDB will not be able to debug user-mode threads: 
 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain 
 conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB. Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".
 doadump () at pcpu.h:159
 (kgdb) bt full
 #0 doadump () at pcpu.h:159
 No locals.
 #1 0xc04f44a6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:397
 first_buf_printf = 1
 #2 0xc04f4854 in panic (fmt=0xc0648063 "%s") at 
 /usr/src/sys/kern/kern_shutdown.c:553
 td = (struct thread *) 0xc1ef0190
 bootopt = 260
 newpanic = 0
 ap = 0xc1ef0190 " ЧНа╟\034Ха"
 buf = "page fault", '\0' <repeats 245 times>
 #3 0xc062a951 in trap_fatal (frame=0xe4a98be4, eva=0) at 
 /usr/src/sys/i386/i386/trap.c:809
 code = 16
 type = 12
 ss = 16
 esp = 0
 softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27, ssd_dpl = 
 0, ssd_p = 1, ssd_xx = 8, ssd_xx1 = 0, ssd_def32 = 1,
 ssd_gran = 1}
 #4 0xc062a648 in trap_pfault (frame=0xe4a98be4, usermode=0, eva=8) at 
 /usr/src/sys/i386/i386/trap.c:727
 va = 0
 vm = (struct vmspace *) 0x0
 map = 0xc0699540
 rv = 1
 ftype = 1 '\001'
 td = (struct thread *) 0xc1ef0190
 p = (struct proc *) 0xc1eefe20
 #5 0xc062a22d in trap (frame=
 {tf_fs = -983629800, tf_es = -1037172720, tf_ds = -458686448, tf_edi = 
 -1032472574, tf_esi = -1040689152, tf_ebp = -458650556, tf_isp = 
 -458650608, tf_ebx = 2048, tf_edx = 64, tf_ecx = 1514, tf_eax = 0, 
 tf_trapno = 12, tf_err = 0, tf_eip = -1068069940, tf_cs = 8, tf_eflags = 
 66182, tf_esp = -1003791136, tf_ss = 32}) at 
 /usr/src/sys/i386/i386/trap.c:417
 td = (struct thread *) 0xc1ef0190
 p = (struct proc *) 0xc1eefe20
 sticks = 0
 i = 0
 ucode = 0
 type = 12
 code = 0
 eva = 8
 #6 0xc0616c2a in calltrap () at /usr/src/sys/i386/i386/exception.s:140
 No locals.
 #7 0xc55f0018 in ?? ()
 No symbol table info available.
 #8 0xc22e0010 in ?? ()
 No symbol table info available.
 #9 0xe4a90010 in ?? ()
 No symbol table info available.
 #10 0xc275b802 in ?? ()
 No symbol table info available.
 #11 0xc1f85800 in ?? ()
 No symbol table info available.
 #12 0xe4a98c44 in ?? ()
 No symbol table info available.
 #13 0xe4a98c10 in ?? ()
 No symbol table info available.
 #14 0x00000800 in ?? ()
 No symbol table info available.
 #15 0x00000040 in ?? ()
 No symbol table info available.
 #16 0x000005ea in ?? ()
 ---Type <return> to continue, or q <return> to quit---
 No symbol table info available.
 #17 0x00000000 in ?? ()
 No symbol table info available.
 #18 0x0000000c in ?? ()
 No symbol table info available.
 #19 0x00000000 in ?? ()
 No symbol table info available.
 #20 0xc0568bcc in bpf_mtap (bp=0x0, m=0xc2740c00) at 
 /usr/src/sys/net/bpf.c:1214
 d = (struct bpf_d *) 0x800
 pktlen = 3262494722
 slen = 0
 #21 0xc056fafd in ether_input (ifp=0xc1f85800, m=0xc2740c00) at 
 /usr/src/sys/net/if_ethersubr.c:543
 eh = (struct ether_header *) 0xc275b802
 etype = 2048
 #22 0xc0480cd1 in em_process_receive_interrupts (adapter=0xc1f85800, 
 count=-5) at /usr/src/sys/dev/em/if_em.c:2867
 ifp = (struct ifnet *) 0xc1f85800
 mp = (struct mbuf *) 0xc42b5ce0
 accept_frame = 1 '\001'
 eop = 1 '\001'
 len = 64
 desc_len = 0
 prev_len_adj = 0
 i = 3251
 current_desc = (struct em_rx_desc *) 0xe4c2cb30
 #23 0xc047d9ff in em_intr (arg=0xc1f85800) at 
 /usr/src/sys/dev/em/if_em.c:1016
 loop_cnt = 3
 reg_icr = 0
 ifp = (struct ifnet *) 0xc1f85800
 adapter = (struct adapter *) 0xc1f85800
 #24 0xc04dc01f in ithread_loop (arg=0xc1e78a00) at 
 /usr/src/sys/kern/kern_intr.c:547
 ithd = (struct ithd *) 0xc1e78a00
 ih = (struct intrhand *) 0xc1fad940
 td = (struct thread *) 0xc1ef0190
 p = (struct proc *) 0xc1eefe20
 count = 0
 warming = 0
 warned = 0
 #25 0xc04dadb3 in fork_exit (callout=0xc04dbe7b <ithread_loop>, arg=0x0, 
 frame=0x0) at /usr/src/sys/kern/kern_fork.c:811
 p = (struct proc *) 0xc1eefe20
 td = (struct thread *) 0x0
 #26 0xc0616c8c in fork_trampoline () at 
 /usr/src/sys/i386/i386/exception.s:209
 No locals.
 
 

From: Maxim Konovalov <maxim@FreeBSD.org>
To: Vladimir Ivanov <wawa@yandex-team.ru>
Cc: rwatson@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Tue, 9 Nov 2004 23:35:41 +0300 (MSK)

 On Tue, 9 Nov 2004, 20:10-0000, Vladimir Ivanov wrote:
 
 > The following reply was made to PR kern/73719; it has been noted by GNATS.
 >
 > From: Vladimir Ivanov <wawa@yandex-team.ru>
 > To: freebsd-gnats-submit@FreeBSD.org, wawa@yandex-team.ru
 > Cc:
 > Subject: Re: kern/73719: Page fault in bpf_mtap ()
 > Date: Tue, 09 Nov 2004 23:02:26 +0300
 >
 >  Ok,
 >  The bpf_mtap () seems to be little enough to make a look.
 >
 >  We suppose that most probable reason to panic is zero value of "bp" pointer.
 >  Also, I know that bpf open/close are frequent on my system.
 >  We can see (look at BPF_MTAP definition) that the value may be changed
 >  from another thread after verification but before bpf_mtap call because
 >  "ifp" points to global variable. The patch does not change the logic of
 >  program as you can see but garantee "bp" is not NULL. The only side
 >  effect is hypotetic pushing extra packet to just detached bpf device.
 >  It's not very big price I seem
 >
 >  I've commited the patch to the system and awaiting results.
 
 I can't believe this technique is a right way to fix things.
 
 Robert, is bpf(4) MP safe already?
 
 -- 
 Maxim Konovalov
Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Thu Nov 11 20:15:15 GMT 2004 
Responsible-Changed-Why:  
Grab ownership of this PR.  BPF was locked down prior to my starting 
work on network stack locking; I've corrected some bugs in the locking 
but it looks like others may remain.  I'll take a look and see what I 
come up with.  My belief was that the locking was correct in the 
presence of descriptors coming and going, but might be less safe with 
interfaces coming and going.  Do interfaces appear and diappear in this 
system, or just descriptors? 


http://www.freebsd.org/cgi/query-pr.cgi?pr=73719 

From: Vladimir Ivanov <wawa@yandex-team.ru>
To: Maxim Konovalov <maxim@FreeBSD.org>
Cc: rwatson@FreeBSD.org, bug-followup@FreeBSD.org
Subject: Re: kern/73719: Page fault in bpf_mtap ()
Date: Fri, 19 Nov 2004 11:30:37 +0300

 Responsible-Changed-From-To: freebsd-bugs->rwatson 
 Responsible-Changed-By: rwatson 
 Responsible-Changed-When: Thu Nov 11 20:15:15 GMT 2004 
 Responsible-Changed-Why:  
 Grab ownership of this PR.  BPF was locked down prior to my starting 
 work on network stack locking; I've corrected some bugs in the locking 
 but it looks like others may remain.  I'll take a look and see what I 
 come up with.  My belief was that the locking was correct in the 
 presence of descriptors coming and going, but might be less safe with 
 interfaces coming and going.  Do interfaces appear and diappear in this 
 system, or just descriptors? 
 
 
 Hi, Robert
 
 I've strarted tcpdump from cron several times per hour. The system 
 carries a huge network traffic.
 Supposed panic scenario:
 
 Network:                                                  User space:
 
                                                                   
 tcpdump started ....
                                                                   bpf 
 attached to network interface
 
 packet occured
 
 we have found bpf pointer in
 interface descriptor, so we have
 to push packet to bpf
 
                                                                    
 tcpdump finished
                                                                    bpf 
 detached
                                                                    
 appropriate reference in network interface descriptor has been cleared
 
 
 reading bpf pointer from network descriptor once more (!!!)
 dereferencing bpf pointer (w/o additional verification !)
 PANIC
 
 
 
 Pls, note: The proposed patch doesn't lock the code flow (It's not a 
 good idea to make a lock per each packet I think).
 
 
 Maxim Konovalov wrote:
 
 >On Tue, 9 Nov 2004, 20:10-0000, Vladimir Ivanov wrote:
 >
 >  
 >
 >>The following reply was made to PR kern/73719; it has been noted by GNATS.
 >>
 >>From: Vladimir Ivanov <wawa@yandex-team.ru>
 >>To: freebsd-gnats-submit@FreeBSD.org, wawa@yandex-team.ru
 >>Cc:
 >>Subject: Re: kern/73719: Page fault in bpf_mtap ()
 >>Date: Tue, 09 Nov 2004 23:02:26 +0300
 >>
 >> Ok,
 >> The bpf_mtap () seems to be little enough to make a look.
 >>
 >> We suppose that most probable reason to panic is zero value of "bp" pointer.
 >> Also, I know that bpf open/close are frequent on my system.
 >> We can see (look at BPF_MTAP definition) that the value may be changed
 >> from another thread after verification but before bpf_mtap call because
 >> "ifp" points to global variable. The patch does not change the logic of
 >> program as you can see but garantee "bp" is not NULL. The only side
 >> effect is hypotetic pushing extra packet to just detached bpf device.
 >> It's not very big price I seem
 >>
 >> I've commited the patch to the system and awaiting results.
 >>    
 >>
 >
 >I can't believe this technique is a right way to fix things.
 >
 >Robert, is bpf(4) MP safe already?
 >
 >  
 >

Adding to audit trail:

See also kern/87014, which although a duplicate, contains more
discussion and a proposed solution.
Responsible-Changed-From-To: rwatson->csjp 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Mon Jun 12 11:42:57 UTC 2006 
Responsible-Changed-Why:  
Assign PR to csjp, who has recently been working on locking issues in 
BPF, and may have fixed this as part of his recent work. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=73719 
State-Changed-From-To: open->patched 
State-Changed-By: csjp 
State-Changed-When: Mon Jun 12 14:56:27 UTC 2006 
State-Changed-Why:  
This should be fixed in HEAD, and will be merged to -STABLE 
after a suffucient testing period expires. 

Thanks. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73719 
State-Changed-From-To: patched->closed 
State-Changed-By: csjp 
State-Changed-When: Sun Jan 21 02:58:38 UTC 2007 
State-Changed-Why:  
The fix has been MFCed 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73719 
>Unformatted:
