            Open Fabrics Enterprise Distribution (OFED)
                CHELSIO T3 RNIC RELEASE NOTES
			January 2008



Author: Steve Wise

The iw_cxgb3 and cxgb3 modules provide RDMA and NIC support for the
Chelsio S310/320 and R310/320 series adapters.  Make sure you choose the
'cxgb3' and 'libcxgb3' options when generating your ofed-1.3 rpms.

============================================
Loadable Module options:
============================================

The following options can be used when loading the iw_cxgb3 module to
tune the iWARP driver:

cong_flavor     - set the congestion congtrol algorithm.  Default is 1.
                  0 == Reno
                  1 == Tahoe
                  2 == NewReno
                  3 == HighSpeed

snd_win         - set the TCP send window in bytes. Default is 32KB.

rcv_win         - set the TCP receive window in bytes. Default is 256KB.

crc_enabled     - set whether MPA CRC should be negotiated.  Default is 1.

markers_enabled - set whether to request receiving MPA markers.  Default is
                  0; do not request to receive markers.

                  NOTE: The Chelsio RNIC fully supports markers, but
                  the current OFA RDMA-CM doesn't provide an API for
                  requesting either markers or crc to be negotiated.  Thus
                  this functionality is provided via module parameters.

mpa_rev         - set the MPA revision to be used.  Default is 1, which is 
                  spec compliant.  Set to 0 to connect with the Ammasso 1100 
                  rnic.

ep_timeout_secs - set the number of seconds for timing out MPA start up
                  negotiation and normal close.  Default is 10.

The following options can be used when loading the cxgb3 module to
tune the NIC driver:

msi             - whether to use MSI or MSI-X.  Default is 2.
                  0 = only pin
                  1 = only MSI or pin
                  2 = use MSI/X, MSI, or pin, based on system

============================================
Updating Firmware:
============================================

This release requires firmware version 5.x, and Protocol SRAM version
1.1.x.  This firmware can be downloaded from http://service.chelsio.com.

If your distro/kernel supports firmware loading, you can place the
chelsio firmware and psram images in /lib/firmware, then unload and reload
the cxgb3 module to get the new images loaded.  If this does not work,
then you can load the firmware images manually:

Obtain the cxgbtool tool and the update_eeprom.sh script from Chelsio.

To build cxgbtool:

# cd <path-to-cxgbtool>
# make && make install

Then load the cxgb3 driver:

# modprobe cxgb3

Now note the ethernet interface name for the T3 device.  This can be
done by typing 'ifconfig -a' and noting the interface name for the
interface with a HW address that begins with "00:07:43".  Then load the
new firmware and eeprom file:

# cxgbtool ethxx loadfw <firmware_file>
# update_eeprom.sh ethxx <eeprom_file>
# reboot

============================================
Testing connectivity with ping and rping:
============================================

Configure the ethernet interfaces for your cxgb3 device.   After you
modprobe iw_cxgb3 you will see one or two ethernet interfaces for the
T3 device.  Configure them with an appropriate ip address, netmask, etc.
You can use the Linux ping command to test basic connectivity via the
T3 interface.

To test RDMA, use the rping command that is included in the librdmacm-utils 
rpm:

On the server machine:

# rping -s -a 0.0.0.0 -p 9999

On the client machine:

# rping -c -VvC10 -a server_ip_addr -p 9999

You should see ping data like this on the client:

ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
client DISCONNECT EVENT...
#

============================================
Addition Notes and Issues
============================================

1) To run uDAPL over the chelsio device, you must export this environment
variable:

        export DAPL_MAX_INLINE=64

2) If you have a multi-homed host and the physical ethernet networks
are bridged, or if you have multiple chelsio rnics in the system, then
you need to configure arp to only send replies on the interface with
the target ip address:

        sysctl -w net.ipv4.conf.all.arp_ignore=2

3) If you are building OFED against a kernel.org kernel later than
2.6.20, then make sure your kernel is configured with the cxgb3 and
iw_cxgb3 modules enabled.  This forces the kernel to pull in the genalloc
allocator, which is requried for the OFED iw_cxgb3 module.  Make sure
these config options are included in your .config file:

CONFIG_CHELSIO_T3=m
CONFIG_INFINIBAND_CXGB=m
