            Open Fabrics Enterprise Distribution (OFED)

                Emulex OCe14000 Adapter RELEASE NOTES

			     June 2015





The ocrdma and be2net modules provide RDMA and NIC support for the

Emulex OCe14000 family of adapters. Make sure you choose the 'ocrdma'

and 'libocrdma' options while building OFED rpms.



============================================

New for OFED-3.18

============================================

- Bug Fixes in ocrdma and libocrdma packages

===========================

Supported Operating Systems

===========================

  o   CPU architectures:

	- x86_64

	- x86

  o   Linux Operating Systems:


	- RedHat EL6.5 		2.6.32-431.el6

	- RedHat EL6.6		2.6.32-504.el6
	
	- RedHat EL7.0		3.10.0-123.el7

	- RedHat EL7.1		3.10.0-229.el7

	- SLES11 SP3		3.0.76-0.9.1

	- SLES12		3.12.28-4-default

	- kernel.org		3.18 *



      * Minimal QA for these versions.

================

Supported Cards

================

The OCe14000 family of adapters are supported.

=========================================

Priority Flow Control recommendations

=========================================

- If the HCA port is connected to a PFC enabled switch, the OCe14000 adapter
  is programmed to use priority 5. Configure the priority-groups
  and switch ports accordingly.

- If the HCA port is connected to a PFC enabled switch, it is
  highly recommended that you use a VLAN interface for traffic. On a non-VLAN 
  interface driver would insert VID 0 for all outgoing packets.

- In back-to-back configurations of OCe14000 series, PFC is enabled by default.



====================

Errata (Open Issues)

====================


- For a 40Gb adapter, ibv_devinfo -v will display:

  active_width:           4X (2)

  active_speed:           10.0 Gbps (4)


  4x * 10 = 40G

  link-width is not applicable to RoCE. ibv_devinfo needs a fix.


=================

Updating Firmware

=================


Firmware version 10.2.x or greater  includes new features that required new

flash regions to support them. Firmware versions earlier than 10.0.803.37

did not have the ability to configure the flash regions to support these 

new features.



Note 1: Do not use any inbox Linux drivers to update from any version of 

	10.0.x firmware to any version of 10.2.x  or greater firmware.


Note 2: If you are using out-of-box Linux drivers and you are updating 

	from a firmware version earlier than 10.0.803.37, you must first 

	update the out-of-box Linux driver to version 10.2.x, and then 

	perform the firmware update procedure twice.


Note 3: If you are using out-of-box Linux drivers and you are updating 

	from firmware version 10.0.803.37 or later, you must first update 

	the out-of-box Linux driver to version 10.2.x. It is not necessary 

	to perform the firmware update procedure twice.



Note 4: After you have updated the firmware, you must not downgrade it to 

	a version earlier than 10.0.803.37.



- Get oc14-10.2.XX.X.ufi or later firmware from www.emulex.com.

- Copy the firmware file into /lib/firmware

- Run: ethtool -f ethX <fw-image-filename>

- Reboot the machine.



The driver in this distribution has been tested with versions 10.6.88.0 and
10.6.103.8

eg:

# cp <path-to>/oc14-10.6.88.0.ufi /lib/firmware/

# ethtool -f eth2 oc14-10.6.88.0.ufi

# reboot


=============

OpenMPI:

=============

From the OFED-3.5 package onwards, OpenMPI is not a part of the OFED

package. You must manually download and install it.



- Download the latest OpenMPI released SRPM from the following location.

  http://www.open-mpi.org/software/ompi



- Build OpenMPI using the following steps:

  ./configure --with-openib

  make

  make install



- Use the following parameters while running an MPI application:

  --mca btl openib,self,sm --mca pml ob1 --mca btl_openib_flags 306 --mca

  btl_openib_receive_queues P,65536,120,64,32 --mca btl_openib_cpc_include

  rdmacm



  eg:

  mpirun -np 8 -hostfile <path to hostfile> --mca btl openib,self,sm --mca pml ob1 \

	--mca btl_openib_flags 306 --mca btl_openib_receive_queues P,65536,120,64,32 \

	--mca btl_openib_cpc_include rdmacm <path to MPI application>



- For further details refer to the OpenMPI documentation.



=============

MVAPICH2:

=============

From the OFED-3.5 package onwards, MVAPICH2 is not a part of the OFED

package. You must manually download and install it.



- Download the latest MVAPICH2 released SRPM from the following location:

  http://mvapich.cse.ohio-state.edu/download/mvapich2/



- Build MVAPICH2 with the following configure options

  ./configure --prefix=/usr --with-device=ch3:mrail --with-rdma=gen2 --sysconfdir=/etc --libdir=/usr/lib64

  make

  make install

- Use the following -env options to run the MPI job in the mpirun command:

  "-env MV2_USE_RDMAOE 1 -env MV2_USE_RDMA_CM 1 -env MV2_USE_SHARED_MEM 1 

  -env MV2_USE_BLOCKING 0 -env MV2_IBA_HCA ocrdma0"

  eg:

  mpirun -np 8 -f <Path to hostfile> -env MV2_USE_RDMAOE 1 -env MV2_USE_RDMA_CM 1 \

	-env MV2_DEFAULT_MAX_CQ_SIZE 128 -env MV2_USE_SRQ 0 -env MV2_MX_SEND_WR 64 \

	-env MV2_MAX_RECV_WR 64 -env MV2_USE_SHARED_MEM 1 -env MV2_USE_BLOCKING 0 \

	-env MV2_IBA_HCA ocrdma0 <path to MPI application>



- For further details refer to the MVAPICH2 documentation.



============================================

Testing connectivity with ping and rping:

============================================



- modprobe be2net

- Configure the Ethernet interfaces for the Emulex device.

- Configure the IP address to the interfaces.

- modprobe ocrdma

- ibv_devinfo should display the ocrdma device in an active state.

- ping <IP of peer Emulex device> should succeed.



Test RDMA connectivity, using the rping command. It is included in the

librdmacm-utils rpm:



On the server machine:



# rping -s -p 9999



On the client machine:



# rping -c -VvC10 -a server_ip_addr -p 9999



You should see ping data like this on the client:



ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr

ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs

ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst

ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu

ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv

ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw

ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx

ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy

ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz

ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA

client DISCONNECT EVENT...





