From nobody@FreeBSD.org  Fri Jan 16 06:09:24 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B7163106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 16 Jan 2009 06:09:24 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 8AF378FC17
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 16 Jan 2009 06:09:24 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n0G69OtU038702
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 16 Jan 2009 06:09:24 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n0G69O6b038695;
	Fri, 16 Jan 2009 06:09:24 GMT
	(envelope-from nobody)
Message-Id: <200901160609.n0G69O6b038695@www.freebsd.org>
Date: Fri, 16 Jan 2009 06:09:24 GMT
From: Jason Chambers <jchambers@ucla.edu>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Certain hardware produces "Network is unreachable" errors for scanning tools
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         130605
>Category:       kern
>Synopsis:       [tcp] Certain hardware produces "Network is unreachable" errors for scanning tools
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-net
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jan 16 06:10:00 UTC 2009
>Closed-Date:    Wed Mar 11 15:12:11 UTC 2009
>Last-Modified:  Wed Mar 11 15:12:11 UTC 2009
>Originator:     Jason Chambers
>Release:        7.1
>Organization:
UCLA
>Environment:
FreeBSD box 7.1-RELEASE-p2 FreeBSD 7.1-RELEASE-p2 #3: Thu Jan  15 21:39:37 PST 2009     user@box:/usr/obj/usr/src/sys/UCLA7-64  amd64
>Description:
On certain hardware platforms FreeBSD has problems issuing multiple
outbound requests as typically seen with network security auditing tools.

The current hardware platform in question is a Dell PowerEdge SC1435.
I have 4 of these servers and all exhibit the same symptoms. 

As an example, NMAP or Nessus fails to perform reconnaissance.  Nessus
does not produce much for errors; numerous "BIOCSRTIMEOUT: Invalid
argument" pcap related error lines are found in the nessusd.dump log.
The "scan" finishes within 6 seconds with the conclusion that all hosts
are dead.  The same Nessus build works without incident on other hardware
platforms running the same version of FreeBSD.  Using TCPDUMP I can see
that requests are being sent on the network and the associated replies
are being returned.

Nmap is unable to scan a system because it immediately goes into a loop
of the following:

  sendto in send_ip_packet: sendto(4, packet, 44, 0,
   xxx.xxx.xxx.xxx, 16) => Network is unreachable

  Offending packet: TCP xxx.xxx.xxx.xxx:55555 > xxx.xxx.xxx.xxx:80
   S ttl=56 id=xxxx iplen=11264  seq=xxxx win=1024 <mss 1460>

  Sleeping 15 seconds then retrying

  sendto in send_ip_packet: sendto(4, packet, 44, 0,
   xxx.xxx.xxx.xxx, 16) => Network is unreachable

  Offending packet: TCP xxx.xxx.xxx.xxx:55555 > xxx.xxx.xxx.xxx:80
   S ttl=56 id=xxxx iplen=11264  seq=xxxx win=1024 <mss 1460>

  Sleeping 60 seconds then retrying

The same "Network is unreachable" messages appear if I attempt to ping
a host while a Nessus scan is running.

I found a PR from a long time ago (2006) that appears relevant but it was
abandoned.  Making updates to that PR does not kick off an email to the
FreeBSD-bugs mailing list so I made the conclusion that it is not working
with GNATS correctly and decided to open a new PR.

http://www.freebsd.org/cgi/query-pr.cgi?pr=102741&cat=

[bugmeister note: his followup is now showing up in 102741]

I tried using a different network controller (bge vs em) but this had no
effect.

I have dedicated one of these systems to troubleshooting this problem if
anyone would like to provide pointers towards further debugging.  

Regards,

--Jason



>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Jan 16 08:22:56 UTC 2009 
Responsible-Changed-Why:  
I'm guessing this is a driver problem rather than 'tcp', but I'll use 
that as a placeholder for now. 

Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=130605 
State-Changed-From-To: open->feedback 
State-Changed-By: rwatson 
State-Changed-When: Mon Feb 2 13:32:50 UTC 2009 
State-Changed-Why:  
Hi Jason: 

Thanks for your detailed bug report.  It seems like a few things are going 
on here, and probably need to be diagnosed individaully.  First, the error 
reported by Nessus, "BIOCSRTIEOUT: Invalid argument" can, I believe, only 
be triggered in the following kernel code: 

int 
itimerfix(struct timeval *tv) 
{ 

if (tv->tv_sec < 0 || tv->tv_usec < 0 || tv->tv_usec >= 1000000) 
return (EINVAL); 
if (tv->tv_sec == 0 && tv->tv_usec != 0 && tv->tv_usec < tick) 
tv->tv_usec = tick; 
return (0); 
} 

This suggests that Nessus is passing an unexpectedly high or low number 
of usec's, and is therefore probably an application bug. 

In general, "Network is unreachable" (ENETUNREACH) is generated by protocol 
sockets when the destination host is on a non-local network and the gateway 
specified in the route to the host is unreachable -- for example, ARP can't 
find the gateway, the device link is down, etc. 

Is there any indication in the system logs of the link state going up and 
down?  You can use "route -n monitor" to track some of the relevant events. 
Given that you've tried multiple cards, I can't help but wondering if 
there is a cabling, switch, or router problem, so if you haven't already, 
I'd follow those possible lines of diagnosis as well. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=130605 

From: Jason Chambers <jchambers@ucla.edu>
To: rwatson@FreeBSD.org, freebsd-net@FreeBSD.org, bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/130605: [tcp] Certain hardware produces "Network is unreachable"
 errors for scanning tools
Date: Tue, 03 Feb 2009 17:59:35 -0800

 rwatson@FreeBSD.org wrote:
 > Thanks for your detailed bug report.  It seems like a few things are going
 > on here, and probably need to be diagnosed individaully.  First, the error
 > reported by Nessus, "BIOCSRTIEOUT: Invalid argument" can, I believe, only
 > be triggered in the following kernel code:
 
 (...)
 
 > 
 > This suggests that Nessus is passing an unexpectedly high or low number
 > of usec's, and is therefore probably an application bug.
 
 Thanks for pointing this out.  Although unrelated to the issue at hand
 it possibly impacts nessus results and will help push their support team
 in the right direction.  I see this error on a FreeBSD system that
 successfully runs scans.
 
 
 > 
 > In general, "Network is unreachable" (ENETUNREACH) is generated by protocol
 > sockets when the destination host is on a non-local network and the gateway
 > specified in the route to the host is unreachable -- for example, ARP can't
 > find the gateway, the device link is down, etc.
 > 
 > Is there any indication in the system logs of the link state going up and
 > down?  You can use "route -n monitor" to track some of the relevant events.
 > Given that you've tried multiple cards, I can't help but wondering if
 > there is a cabling, switch, or router problem, so if you haven't already,
 > I'd follow those possible lines of diagnosis as well.
 
 
 There's no indication of interface flapping in the logs and I checked
 that the underlying infrastructure is fine.  Route -n monitor shows
 nothing before and during the NMap or Nessus scanning.
 
 I suspect it's hardware related because not all FreeBSD (7.1-p2) systems
 I'm using have the problem.  The commonality in all of it is newer Dell
 rack server systems ( 8-core PowerEdge 1950 and SC1435's).  This
 "Network is unreachable" error seems to always occur with NMAP's OS
 discovery phase.
 
 ex:  nmap -sS -p 22 -O host
 
 However, only on the SC1435's does Nessus fail to run successfully.
 When I say run, I mean the nessus process successfully sends out probe
 traffic (verified at the remote destination) but ignores the replies.
 Tcpdump shows that they arrive fine.  The only thing different about
 these SC1435's from other systems is a patch I've applied to have the
 ServerWorks HT1000 controller work.  (ata_ht1000.patch)
 
 http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2008-10/msg00039.html
 
 
 I converted two of the servers to Linux and everything works fine with
 them.  Rather than bury the problem I'd like to understand what fails.
 
 I can give access to one or both of these machines if it would help the
 effort.  Otherwise, any suggestions on what tests I should run to
 further isolate this problem to a specific subsystem ?
 
 
 --Jason
 

From: Jason Chambers <jchambers@ucla.edu>
To: bug-followup@FreeBSD.org, jchambers@ucla.edu
Cc:  
Subject: Re: kern/130605: [tcp] Certain hardware produces "Network is unreachable"
 errors for scanning tools
Date: Mon, 02 Mar 2009 03:26:43 -0800

 Hello all.
 
 The Nessus problem was finally identified to be caused by incompatible
 ABI system calls.
 
 As for the general "Network is unreachable" error I assume this to be
 due to the condition described or incorrect implementation of said
 security tools on FreeBSD.
 
 Thanks for the help, this PR can be closed.
 
 --Jason
 
State-Changed-From-To: feedback->closed 
State-Changed-By: gavin 
State-Changed-When: Wed Mar 11 15:09:08 UTC 2009 
State-Changed-Why:  
Close, submitter is happy that this has been resolved (see also 132518) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=130605 
>Unformatted:
