From nobody@FreeBSD.org  Wed Feb  7 03:15:32 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id DC82216A400
	for <freebsd-gnats-submit@FreeBSD.org>; Wed,  7 Feb 2007 03:15:30 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id CC63113C441
	for <freebsd-gnats-submit@FreeBSD.org>; Wed,  7 Feb 2007 03:15:30 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l173FUjS007730
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 7 Feb 2007 03:15:30 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l173FUhi007729;
	Wed, 7 Feb 2007 03:15:30 GMT
	(envelope-from nobody)
Message-Id: <200702070315.l173FUhi007729@www.freebsd.org>
Date: Wed, 7 Feb 2007 03:15:30 GMT
From: Lawrence Stewart<lstewart@room52.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: nve(4) driver on FreeBSD 6.2 AMD64 does not work at 1Gbps with nForce4 NIC
X-Send-Pr-Version: www-3.0

>Number:         108861
>Category:       amd64
>Synopsis:       [nve] nve(4) driver on FreeBSD 6.2 AMD64 does not work at 1Gbps with nForce4 NIC
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-amd64
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 07 03:20:18 GMT 2007
>Closed-Date:    Mon Dec 13 10:34:52 UTC 2010
>Last-Modified:  Mon Dec 13 10:34:52 UTC 2010
>Originator:     Lawrence Stewart
>Release:        6.2
>Organization:
Swinburne University
>Environment:
raidserv2# uname -a
FreeBSD raidserv2.caia.swin.edu.au 6.2-RELEASE FreeBSD 6.2-RELEASE #1: Wed Feb  7 11:30:15 EST 2007     root@raidserv2.caia.swin.edu.au:/usr/src/sys/amd64/compile/GENERIC+SMP  amd64

>Description:
The research centre I work at recently purchased a new file server, built from commodity hardware that FreeBSD claimed to have support for.

Motherboard is an Asus A8N-E Rev 2.00 (listed as supported here: http://www.freebsd.org/platforms/amd64/motherboards.html).

I clean installed FreeBSD 6.2 from ISO 6.2-RELEASE-amd64-disc1.iso.

The onboard nVidia nForce4 nve(4) based NIC was plugged into the university's Cisco based gigabit ethernet LAN via CAT5e straight-through patch lead.

An Intel 82540EM em(4) based PCI NIC was also installed in the server, but was not plugged into a network point (we previously tried installing FreeBSD 6.1 which had no support for the onboard NIC, so we put the Intel card in to use. We left it in for the new test with FreeBSD 6.2).

The install completed successfully. We didn't configure either NIC during install at all (well... that's not entriely true the first time through... but I'll save the details of that crazy story for later as it will confuse things for the moment).

After reboot, we bought the NIC up using "ifconfig nve0 up" and it synced at 1000baseTX full-duplex. Tcpdumping on the interface shows no activity at all (even though the uni network is VERY chatty, multiple packets per second should have been showing). Running "dhclient nve0" shows outbound DHCP requests, but not any return packets. 

Forcibly syncing the interface to 1000baseTX using "ifconfig nve0 media 1000baseTX mediaopt full-duplex" resulted in same behaviour i.e. doesn't "see" inbound packets. Forcibly syncing the interface to 100baseTX all of a sudden shows packets being received in tcpdump, and we can use the interface, though it only achieves about 80kb/sec on transfers i.e. is severely restricted for some reason.

Intrigued, I installed a small 5 port 100baseTX switch between the server and network port and set the interface back to autoselect using "ifconfig nve0 media autoselect". The nve(4) interface synced at 100baseTX, and could then transfer at full 100Mbps speed (scp @ 11MB/sec from another local computer).

I tried resolving the issue by setting "hw.pci.enable_msi=0" and "hw.pci.enable_msix=0" in /boot/loader.conf as suggested for fixing em(4) related problems. It had no effect from what I could see. I also tried rebuilding and installing a GENERIC kernel, GENERIC kernel without nve driver and kldloading it after startup, GENERIC+SMP kernel... none of the different kernel configs seemd to change anything.

I also might add that I tried getting the em(4) Intel NIC working when I realised the nve(4) NIC was going to be a pain, and I had no luck, even with the disabling of MSI in /boot/loader.conf. I didn't try any harder than that though because my main focus was seeing if the nve driver was fixed.
>How-To-Repeat:
Patch the onboard ethernet NIC on an Asus A8N-E Rev 2.00 motherboard into a 1GBps switch port.

Install FreeBSD 6.2 from ISO 6.2-RELEASE-amd64-disc1.iso.

Configure the NIC to autoselect it's sync speed using "ifconfig nve0 media autoselect".

Bring the NIC up using "ifconfig nve0 up"

It should sync at 1000baseTX full-duplex, but not be able to communicate at all via the interface.

>Fix:
I could not find a fix for the problem.

I also could not figure out why explicitly setting the NIC to sync at 100baseTX full-duplex made the NIC work, but only at a fraction of 100Mbps capacity.

A work around is to cause the switch port the NIC is plugged into to only allow 100baseTX, so that the NIC will autoselect at 100baseTX and work at reasonable speed. You could also do as I did and stick small 100baseTX switch in between the machine and gigabit switch.
>Release-Note:
>Audit-Trail:

From: "Angelo Turetta" <aturetta+bsd@bestunion.it>
To: lstewart@room52.net
Cc: bug-followup@FreeBSD.org
Subject: Re: amd64/108861: [nve] nve(4) driver on FreeBSD 6.2 AMD64 does 
     not work at 1Gbps with nForce4 NIC
Date: Wed, 7 Feb 2007 15:43:52 +0100 (CET)

 Could you please test the alternative driver mentioned at
 
 http://www.se.hiroshima-u.ac.jp/~shigeaki/software/freebsd-nfe.html
 
 It's already been commited to -current, and I'd certainly hope it will be
 MFC to 6-STABLE soon.
 
 Angelo Turetta
 

From: "Angelo Turetta" <aturetta+bsd@bestunion.it>
To: lstewart@room52.net
Cc: bug-followup@FreeBSD.org
Subject: Re: amd64/108861: [nve] nve(4) driver on FreeBSD 6.2 AMD64 does 
     not work at 1Gbps with nForce4 NIC
Date: Wed, 7 Feb 2007 15:43:58 +0100 (CET)

 Could you please test the alternative driver mentioned at
 
 http://www.se.hiroshima-u.ac.jp/~shigeaki/software/freebsd-nfe.html
 
 It's already been commited to -current, and I'd certainly hope it will be
 MFC to 6-STABLE soon.
 
 Angelo Turetta
 

From: "Scot Hetzel" <swhetzel@gmail.com>
To: "Lawrence Stewart" <lstewart@room52.net>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: amd64/108861: nve(4) driver on FreeBSD 6.2 AMD64 does not work at 1Gbps with nForce4 NIC
Date: Wed, 7 Feb 2007 16:58:36 -0600

 On 2/6/07, Lawrence Stewart <lstewart@room52.net> wrote:
 > I could not find a fix for the problem.
 >
 > I also could not figure out why explicitly setting the NIC to sync at 100baseTX full-duplex made the NIC work, but only at a fraction of 100Mbps capacity.
 >
 > A work around is to cause the switch port the NIC is plugged into to only allow 100baseTX, so that the NIC will autoselect at 100baseTX and work at reasonable speed. You could also do as I did and stick small 100baseTX switch in between the machine and gigabit switch.
 >
 Have you tried forcing the switch switch and the NIC to 1000baseT Full Duplex?
 
 Sometime times auto-negotiation fails, so forcing speed should fix it.
 
 Scot
 -- 
 DISCLAIMER:
 No electrons were mamed while sending this message. Only slightly bruised.

From: Lawrence Stewart <lstewart@room52.net>
To: Angelo Turetta <aturetta+bsd@bestunion.it>
Cc: bug-followup@FreeBSD.org
Subject: Re: amd64/108861: [nve] nve(4) driver on FreeBSD 6.2 AMD64 does 
     not work at 1Gbps with nForce4 NIC
Date: Thu, 08 Feb 2007 13:56:21 +1100

 Hi Angelo,
 
 Thanks very much for the reply.
 
 Angelo Turetta wrote:
 > Could you please test the alternative driver mentioned at
 >
 > http://www.se.hiroshima-u.ac.jp/~shigeaki/software/freebsd-nfe.html
 >
 > It's already been commited to -current, and I'd certainly hope it will be
 > MFC to 6-STABLE soon.
 >
 > Angelo Turetta
 >   
 
 
 The new driver appears to be working perfectly. I'm running it through 
 some basic stress tests now, but everything so far is all good. It can 
 sync at both 100baseTX and 1000baseTX. When in 100baseTX mode, I can get 
 11.1 Mb/sec over scp. When in 1000baseTX I can get 24 MB/sec over scp. 
 This was only transferring from a low spec desktop machine, so I think 
 the machine's disk drive was the bottleneck, not the NIC.
 
 For the benefit of anyone else that comes across this thread and wants 
 to know how to get the new driver working, here are the steps I followed 
 (lines beginning with a hyphen "-" are comments, not actual shell commands):
 
 
 
 cd /root
 
 fetch http://www.se.hiroshima-u.ac.jp/~shigeaki/software/nfe-20070106.tar.gz
 
 
 - You may need a different patch to the one below depending on the 
 motherboard you have... this patch is the right one for my Asus A8N-E, 
 but all the details are on the patch website 
 http://www.se.hiroshima-u.ac.jp/~shigeaki/software/freebsd-nfe.html
 
 fetch 
 http://www.se.hiroshima-u.ac.jp/~shigeaki/software/e1000phy.20061219.fbsd62.patch
 
 tar -xzvf nfe-20070106.tar.gz
 
 cp -r nfe-20070106 /usr/src/sys/dev/nfe
 
 cp e1000phy.20061219.fbsd62.patch /usr/src/sys/dev/mii/
 
 cd /usr/src/sys/dev/mii/
 
 patch < e1000phy.20061219.fbsd62.patch
 
 cd /usr/src/sys/amd64/conf/
 
 - Not sure that you need to do the next step, but I thought it would be 
 safer to remove the nve(4) driver from the kernel so it didn't get 
 confused... you can still kldload it later if needed
 
 edit GENERIC
 
 - Comment out the line "device          nve             # nVidia nForce 
 MCP on-board Ethernet Networking"
 
 
 
 rm -rf ../compile/GENERIC
 
 config GENERIC
 
 cd ../compile/GENERIC
 
 make cleandepend && make depend && make && make install
 
 cd /usr/src/sys/dev/nfe
 
 make
 
 make install
 
 shutdown -r now
 
 
 - You should now have if_nve.ko in /boot/kernel/ and you can kldload it 
 using "kldload if_nfe" or load it at startup by sticking 
 if_nfe_load="YES" in /boot/loader.conf.
 
 - Good times from here on in!
 
 
 
 
 Thanks again for the help.
 
 
 Regards,
 Lawrence Stewart

From: Lawrence Stewart <lstewart@room52.net>
To: Scot Hetzel <swhetzel@gmail.com>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: amd64/108861: nve(4) driver on FreeBSD 6.2 AMD64 does not work
 at 1Gbps with nForce4 NIC
Date: Thu, 08 Feb 2007 16:05:14 +1100

 Hi Scot,
 
 Thanks for the reply.
 
 Scot Hetzel wrote:
 > On 2/6/07, Lawrence Stewart <lstewart@room52.net> wrote:
 >> I could not find a fix for the problem.
 >>
 >> I also could not figure out why explicitly setting the NIC to sync at 
 >> 100baseTX full-duplex made the NIC work, but only at a fraction of 
 >> 100Mbps capacity.
 >>
 >> A work around is to cause the switch port the NIC is plugged into to 
 >> only allow 100baseTX, so that the NIC will autoselect at 100baseTX 
 >> and work at reasonable speed. You could also do as I did and stick 
 >> small 100baseTX switch in between the machine and gigabit switch.
 >>
 > Have you tried forcing the switch switch and the NIC to 1000baseT Full 
 > Duplex?
 >
 > Sometime times auto-negotiation fails, so forcing speed should fix it.
 >
 I don't have access to the actual Cisco switches, as they are maintained 
 by our IT services department. Forcibly setting the switch to 1000 Mbps 
 is kind of besides the point anyway. The problem is that when the NIC is 
 autosensing (default state) and plugged into an autonegotiating 1 Gbps 
 switch port (default for all switches I've ever worked with), the driver 
 appears to be functional for all intensive purposes, except for the fact 
 that no packets can be accessed. That is confusing for the user and is 
 the major problem with the nve(4) driver in its current state. Having to 
 put a call into the IT dept and trying to explain that you need them to 
 manually set the sync speed on a port is unlikely to be well received... 
 it's also simply another work around, rather than a fix.
 
 Regards,
 Lawrence Stewart
 
State-Changed-From-To: open->suspended 
State-Changed-By: obrien 
State-Changed-When: Tue Feb 5 18:00:11 UTC 2008 
State-Changed-Why:  
It is hard to do much with nve due to it being mostly a binary blob. 
In FreeBSD 7.0, one should use nfe(4) instead of nve(4). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=108861 

From: Andriy Gapon <avg@freebsd.org>
To: bug-followup@freebsd.org, lstewart@room52.net
Cc:  
Subject: Re: amd64/108861: [nve] nve(4) driver on FreeBSD 6.2 AMD64 does not
 work at 1Gbps with nForce4 NIC
Date: Sun, 05 Dec 2010 12:01:48 +0200

 Lawrence,
 
 how do you feel about this PR?
 Did David's suggestion work for you?
 
 -- 
 Andriy Gapon

From: Lawrence Stewart <lstewart@room52.net>
To: Andriy Gapon <avg@freebsd.org>
Cc: bug-followup@freebsd.org
Subject: Re: amd64/108861: [nve] nve(4) driver on FreeBSD 6.2 AMD64 does not
 work at 1Gbps with nForce4 NIC
Date: Mon, 13 Dec 2010 15:31:09 +1100

 On 12/05/10 21:01, Andriy Gapon wrote:
 > 
 > Lawrence,
 > 
 > how do you feel about this PR?
 > Did David's suggestion work for you?
 > 
 
 We've been running with nfe on those machines since shortly after I
 logged that PR and the machines have been running (and continue to run)
 stably since. The PR should be closed.
 
 Cheers,
 Lawrence
State-Changed-From-To: suspended->closed 
State-Changed-By: avg 
State-Changed-When: Mon Dec 13 10:34:03 UTC 2010 
State-Changed-Why:  
Closing per originator's request: nfe is a better replacement 
for proprietary nve. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=108861 
>Unformatted:
