From moro@remus.dti.ne.jp  Thu Aug 26 13:36:19 2004
Return-Path: <moro@remus.dti.ne.jp>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 27AD216A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 26 Aug 2004 13:36:19 +0000 (GMT)
Received: from smtp4.dti.ne.jp (smtp4.dti.ne.jp [202.216.228.39])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 3206F43D58
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 26 Aug 2004 13:36:18 +0000 (GMT)
	(envelope-from moro@remus.dti.ne.jp)
Received: from kikyou.m.ayame.com (ntmiex032044.miex.nt.adsl.ppp.infoweb.ne.jp [220.147.92.44]) by smtp4.dti.ne.jp (3.08s) with ESMTP AUTH id i7QDaHXg012732 for <FreeBSD-gnats-submit@freebsd.org>; Thu, 26 Aug 2004 22:36:17 +0900 (JST)
Received: by kikyou.m.ayame.com (Postfix, from userid 1001)
	id DE1775AA2; Thu, 26 Aug 2004 22:36:16 +0900 (JST)
Message-Id: <20040826133616.DE1775AA2@kikyou.m.ayame.com>
Date: Thu, 26 Aug 2004 22:36:16 +0900 (JST)
From: MOROHOSHI Akihiko <moro@remus.dti.ne.jp>
Reply-To: MOROHOSHI Akihiko <moro@remus.dti.ne.jp>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Bug in netisr_queue()
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         70988
>Category:       kern
>Synopsis:       Bug in netisr_queue()
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    andre
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Aug 26 13:40:24 GMT 2004
>Closed-Date:    Thu Sep 16 20:36:45 GMT 2004
>Last-Modified:  Thu Sep 16 20:36:45 GMT 2004
>Originator:     MOROHOSHI Akihiko <moro@remus.dti.ne.jp>
>Release:        FreeBSD 5.3-BETA1 i386
>Organization:
>Environment:
System: FreeBSD kikyou.local.domain 5.3-BETA1 FreeBSD 5.3-BETA1 #6: Thu Aug 26 21:19:55 JST 2004     moro@kikyou.local.domain:/home/tmp/obj/home/releng5/src/sys/KIKYOU  i386

	
>Description:
In short:
netisr_queue() has a bug of its return value.  Please apply the patch.

Background:
When I used ipfw fwd action to realize transparent HTTP proxy,
it worked fine for other machines, but trying "telnet www.foobar.com 80"
on the FreeBSD box resulted in "Operation not permitted."
(Configurations were proven to work fine on 4-stable.
I've been migrating to 5.3-BETA1.)

Analysis:
The reason of the error EPERM(=1) is that ip_output() return 1,
and it is because netisr_queue() return 1.
I observed netisr_queue() always returned 1, even when forwarding
works fine for client machines.

Look at netisr_queue() in sys/net/netisr.c:
| int
| netisr_queue(int num, struct mbuf *m)
| {
| 	struct netisr *ni;
| 	
| 	KASSERT(!(num < 0 || num >= (sizeof(netisrs)/sizeof(*netisrs))),
| 	    ("bad isr %d", num));
| 	ni = &netisrs[num];
| 	if (ni->ni_queue == NULL) {
| 		isrstat.isrs_drop++;
| 		m_freem(m);
| 		return (1);
| 	}
| 	isrstat.isrs_queued++;
| 	if (!IF_HANDOFF(ni->ni_queue, m, NULL))
| 		return (0);
| 	schednetisr(num);
| 	return (1);
| }

In the last line netisr_queue returns 1, but it should return 0.

Also, it seems that netisr_queue assumes IF_HANDOFF returns 0 when succeeded,
but it is wrong. IF_HANDOFF returns 0 when the queue is full, and returns
1 when succeeded:

| #define	IF_HANDOFF(ifq, m, ifp)			\
| 	if_handoff((struct ifqueue *)ifq, m, ifp, 0)

| static __inline int
| if_handoff(struct ifqueue *ifq, struct mbuf *m, struct ifnet *ifp, int adjust)
| {
| 	int active = 0;
| 
| 	IF_LOCK(ifq);
| 	if (_IF_QFULL(ifq)) {
| 		_IF_DROP(ifq);
| 		IF_UNLOCK(ifq);
| 		m_freem(m);
| 		return (0);
| 	}
| 	if (ifp != NULL) {
| 		ifp->if_obytes += m->m_pkthdr.len + adjust;
| 		if (m->m_flags & (M_BCAST|M_MCAST))
| 			ifp->if_omcasts++;
| 		active = ifp->if_flags & IFF_OACTIVE;
| 	}
| 	_IF_ENQUEUE(ifq, m);
| 	IF_UNLOCK(ifq);
| 	if (ifp != NULL && !active)
| 		if_start(ifp);
| 	return (1);
| }

So, this part in netisr_queue() should be changed to return 1:
| 	if (!IF_HANDOFF(ni->ni_queue, m, NULL))
| 		return (0);

	
>How-To-Repeat:

1. Set up squid as a transparent HTTP proxy in jail (192.168.79.21).
2. Add ipfw rules for forwarding:
 ipfw add allow tcp from 192.168.79.21 to any 80 out setup keep-state
 ipfw add fwd 192.168.79.21,8080 tcp from any to any 80 out setup keep-state
3. Browse some web sites with client machines. It should be OK.
4. Try "telnet www.example.com 80" on the FreeBSD box.
   You will see "Operation not permitted."
	
>Fix:
--- netisr.c.1.10	Mon Jul 19 06:50:22 2004
+++ netisr.c	Thu Aug 26 21:06:54 2004
@@ -220,9 +220,9 @@ netisr_queue(int num, struct mbuf *m)
 	}
 	isrstat.isrs_queued++;
 	if (!IF_HANDOFF(ni->ni_queue, m, NULL))
-		return (0);
+		return (1);
 	schednetisr(num);
-	return (1);
+	return (0);
 }
 
 static void

	
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: arved 
Responsible-Changed-When: Thu Aug 26 21:30:57 GMT 2004 
Responsible-Changed-Why:  
Over to freebsd-net for review 

http://www.freebsd.org/cgi/query-pr.cgi?pr=70988 
Responsible-Changed-From-To: freebsd-net->andre 
Responsible-Changed-By: andre 
Responsible-Changed-When: Thu Aug 26 21:36:44 GMT 2004 
Responsible-Changed-Why:  
Take over. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=70988 
State-Changed-From-To: open->patched 
State-Changed-By: andre 
State-Changed-When: Fri Aug 27 18:36:16 GMT 2004 
State-Changed-Why:  
Hello Akihiko-san, 

I have committed your fix plus adjustments to all users of netisr_queue(). 

Revision 1.11 of sys/net/netisr.c contains the fix. 

Many thanks for your very good analysis of the problem! 

--  
Andre 

http://www.freebsd.org/cgi/query-pr.cgi?pr=70988 
State-Changed-From-To: patched->closed 
State-Changed-By: andre 
State-Changed-When: Thu Sep 16 20:36:19 GMT 2004 
State-Changed-Why:  
MFC to RELENG_5 done.  Case closed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=70988 
>Unformatted:
