		Open Fabrics Enterprise Distribution (OFED)
		      Tips for Working with OFED 1.3

		 	     January 2008

===============================================================================
Table of Contents
===============================================================================
1. OFED Utilities
2. Debug HOWTOs
3. Pinning (Locking) User Memory Pages
4. External Module Compilation Over OFED-1.3
5. Adding/Deleting a patch to OFED package
6. Adding vendor specific actions to the installation of OFED


===============================================================================
1. OFED Utilities
===============================================================================

The OFED package includes utilities under <prefix>/bin, where <prefix> stands
for the OFED installation path. To retrieve this path, run the script 
"/etc/infiniband/info" as explained in Section 2.2 below.

Notes:
------
1. This document includes descriptions for a subset of the existing utilities.
   To learn about other utilities, use their --help flag.

2. The sources for all utilities are not part of the RPM installation. However,
   all sources exist in the openib-1.3.tgz tarball.


1.1 Device Information
----------------------
Device information can be obtained using several utilities:

a. ibv_devinfo

   ibv_devinfo prints the ca attributes.

   usage:
     ibv_devinfo

     Options:
       -d, --ib-dev=<dev>    use IB device <dev> (default: first device found)
       -i, --ib-port=<port>  use port <port> of IB device (default: all ports)
       -l, --list            print only the IB devices names
       -v, --verbose         print all the attributes of the IB device(s)

b. ibstat

	usage:
	ibstat [OPTIONS] <ca_name> [portnum]

	Options:
	-d	debug
	-l	list all IB devices
	-s	print short device summary
	-p	print port GUIDs
	-V	print ibstat version information and exit
	-h	print usage

	Examples:
	ibstat -l 		# list all IB devices
	ibstat mthca0 2 	# stat port 2 of mthca0

c. Using sysfs file system
	The driver supports the sysfs file system under: /sys/class/infiniband

	Examples:

	> ls /sys/class/infiniband/mthca0/
	board_id device fw_ver hca_type hw_rev node_desc node_guid node_type
	ports sys_image_guid

	> cat /sys/class/infiniband/mthca0/board_id
	MT_0200000001

	> ls /sys/class/infiniband/mthca0/ports/1/
	cap_mask counters gids lid lid_mask_count phys_state pkeys rate sm_lid
	sm_sl state

	> cat /sys/class/infiniband/mthca0/ports/1/state
	4: ACTIVE

1.2 Performance Tests
---------------------
	The following performance tests are provided with the OFED release:

	1. Latency tests:
	   - ib_read_lat: RDMA read
	   - ib_write_lat: RDMA write
	   - ib_send_lat: UD, UC and RC (default) send

	2. Bandwidth tests:
	   - ib_read_bw: RDMA read
	   - ib_write_bw: RDMA write
	   - ib_send_bw: UD, UC and RC (default) send

	Usage:
	Server: <test name> <options>
	Client: <test name> <options> <server IP address>
		<server IP address> is an Ethernet or IPoIB address.
	--help	  lists the available <options>. The same options must be
		  passed to both server and client.	
	
	Note: See PERF_TEST_README.txt for more information on the performance
	      tests.
	
	Example: ib_send_bw
	Usage:
	ib_send_bw 			start a server and wait for connection
	ib_send_bw <host>		connect to server at <host>

	options:
	-p, --port=<port> 		listen on/connect to port <port>
						(default: 18515)
	-d, --ib-dev=<dev> 		use IB device <dev>
						(default: first device found)
	-i, --ib-port=<port> 		use port <port> of IB device
						(default: 1)
	-c, --connection=<RC/UC/UD> 	connection type RC/UC/UD (default: RC)
	-m, --mtu=<mtu> 		mtu size (default: 1024)
	-s, --size=<size> 		size of message to exchange
						(default: 65536)
	-a, --all 			run sizes from 2 up to 2^23
	-t, --tx-depth=<dep> 		size of tx queue (default: 300)
	-n, --iters=<iters> 		number of exchanges
						(at least 2, default: 1000)
	-b, --bidirectional 		measure bidirectional bandwidth
						(default: unidirectional)
	-V, --version display version number

1.3 Ping-pong Example Tests
---------------------------
	The ping-pong example tests provide basic connectivity tests. Each test
	has a help message (-h).
	- ibv_ud_pingpong
	- ibv_rc_pingpong
	- ibv_srq_pingpong
	- ibv_uc_pingpong

	Example: ibv_ud_pingpong --h
	Usage:
	ibv_ud_pingpong  	start a server and wait for connection
	ibv_ud_pingpong <host>	connect to server at <host>

	options:
	-p, --port=<port> 	listen on/connect to port <port>
					(default: 18515)
	-d, --ib-dev=<dev> 	use IB device <dev>
					(default: first device found)
	-i, --ib-port=<port> 	use port <port> of IB device (default: 1)
	-s, --size=<size> 	size of message to exchange (default: 2048)
	-r, --rx-depth=<dep> 	number of receives to post at a time
					(default: 500)
	-n, --iters=<iters> 	number of exchanges (default: 1000)
	-e, --events 		sleep on CQ events (default: poll)


===============================================================================
2. Debug HOWTOs
===============================================================================

2.1 OFED Components and Version Information
-------------------------------------------
The text file BUILD_ID provides data on all OFED components (whether installed
or not). This file is a part of the ofed-docs RPM and installed under 
/usr/share/doc/ofed-docs-1.3 on RedHat, and under 
/usr/share/doc/packages/ofed-docs-1.3 on SuSE.

The same information can be obtained by executing the 'ofed_info' command. For
example:

> ofed_info
OFED-1.3-20080128-0600

ofa_kernel-1.3:
Git:
git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel
commit 359f200e73049284f5ee9fe545de226f9f902039

libibverbs:
git://git.openfabrics.org/ofed_1_3/libibverbs.git ofed_1_3
commit 6e99cddf835d4715ea7ca3641944e6285f27f2df
libmthca:
git://git.openfabrics.org/ofed_1_3/libmthca.git ofed_1_3
commit 9501e698d257949acfab2edc90812602966dbcc9
libmlx4:
git://git.openfabrics.org/ofed_1_3/libmlx4.git ofed_1_3
commit 3869d6dab7e12fe452270ca641f7dd7082b42482
libehca:
git://git.openfabrics.org/ofed_1_3/libehca.git ofed_1_3
commit dbcba8e758fcccfa1727821bd0da64a44105a425
libipathverbs:
git://git.openfabrics.org/ofed_1_3/libipathverbs.git ofed_1_3
commit 306d414eab5f62a959746e427b0af02676007c45
libcxgb3:
git://git.openfabrics.org/ofed_1_3/libcxgb3.git ofed_1_3
commit 946e1572d38d786038d3bca5a2ec184dcf7636c4
libnes:
git://git.openfabrics.org/ofed_1_3/libnes.git ofed_1_3
commit 27ebf415cb65918237c7c21fd8b28bc1dbf4fca6
libibcm:
git://git.openfabrics.org/~shefty/libibcm.git master
commit a45e43483ac29a26c1803f217ca21a07534494c4
librdmacm:
git://git.openfabrics.org/ofed_1_3/librdmacm.git ofed_1_3
commit afe87c16f40fe4a3622f231672737950c0ebf9fa
dapl1:
git://git.openfabrics.org/~ardavis/dapl.git master
commit 95935648f81ac5177ae7120b35e55483902b8c64
dapl2:
git://git.openfabrics.org/~ardavis/dapl.git dat2.0
commit abb4356cd765d38cf5cff2dfbdb610b380f944a2
libsdp:
git://git.openfabrics.org/ofed_1_3/libsdp.git ofed_1_3
commit 47801f8f1e2168c34690b93edaccadc2ece936ef
sdpnetstat:
git://git.openfabrics.org/ofed_1_3/sdpnetstat.git ofed_1_3
commit 3341620a7259c4f7bdd4180864b98e260c3dc223
srptools:
git://git.openfabrics.org/ofed_1_3/srptools.git ofed_1_3
commit d127b2ed8280aaccd8c071212388255040221c33
perftest:
git://git.openfabrics.org/ofed_1_3/perftest.git ofed_1_3
commit a909b982fa6494a179faaf817be3ef019e106199
qlvnictools:
git://git.openfabrics.org/ofed_1_3/qlvnictools.git ofed_1_3
commit 41a148393a602810df80109e71086970a91c1d8d
tvflash:
git://git.openfabrics.org/~rdreier/tvflash.git master
commit 39a63301f0344b6b3d45bc4b16d76be81f4377c0
mstflint:
git://git.openfabrics.org/ofed_1_3/mstflint.git ofed_1_3
commit 260ae655d80fa3ed15c4544519a4c7f0cf768bd6
qperf:
git://git.openfabrics.org/ofed_1_3/qperf.git ofed_1_3
commit f6863fac0fd55659aa3a9acc9b47f7d5eb6f6e22
management:
git://git.openfabrics.org/ofed_1_3/management.git ofed_1_3
commit 264eb24f3b2b05dd1ee2f5c1025470845cf8ca36
ibutils:
git://git.openfabrics.org/ofed_1_3/ibutils.git ofed_1_3
commit 0225143c82416d02d6f00cf93bb0f38915557a12
ibsim:
git://git.openfabrics.org/~sashak/ibsim.git master
commit 3b20040b50da6afc5c4ef85f6d458f9023ed7ed9

# MPI
mvapich-1.0.0-1870.src.rpm
mvapich2-1.0.1-2.src.rpm
openmpi-1.2.5-1.src.rpm
mpitests-3.0-773.src.rpm

2.2 Installed OFED Components
-------------------------------
The script /etc/infiniband/info provides data on the specific OFED installation
on the machine.

For example:

 > /etc/infiniband/info
   prefix=/usr
   Kernel=2.6.9-22.ELsmp

   MODULES:  CONFIG_INFINIBAND=m CONFIG_INFINIBAND_USER_MAD=m
   CONFIG_INFINIBAND_USER_ACCESS=m CONFIG_INFINIBAND_ADDR_TRANS=y
   CONFIG_INFINIBAND_MTHCA=m CONFIG_IPATH_CORE=m CONFIG_INFINIBAND_IPATH=m
   CONFIG_INFINIBAND_IPOIB=m

   User level:  --kernel-version 2.6.9-22.ELsmp --kernel-sources
   /lib/modules/2.6.9-22.ELsmp/build --with-libibcm --with-libibverbs
   --with-libipathverbs --with-libmthca --with-mstflint --with-perftest

2.3 Building/Installing InfiniBand (IB) Modules With Debug Information
----------------------------------------------------------------------
To compile/build/install the IB modules so that they will contain debug
information, set OPENIB_KERNEL_EXTRA_CFLAGS="-g" in your environment
before running OFED's install.sh/build.sh .


===============================================================================
3. Pinning (Locking) User Memory Pages
===============================================================================

Memory locking is managed by the kernel on a per user basis. Regular users (as
opposed to root) have a limited number of pages which they may pin, where
the limit is pre-set by the administrator. Registering memory for IB verbs
requires pinning memory, thus an application cannot register more memory than
it is allowed to pin.

The user can change the system per-process memory lock limit by adding
the following two lines to the file /etc/security/limits.conf:

  *  soft  memlock  <number>
  *  hard  memlock  <number>

  where <number> denotes the number of KBytes that may be locked by a
  user process.

The above change to /etc/security/limits.conf will allow any user process in the
system to lock up to <number> KBytes of memory.

On some systems, it may be possible to use "unlimited" for the size to disable
these limits entirely.
	      
Note: The file /etc/security/limits.conf contains further documentation.


===============================================================================
4. External Module Compilation Over OFED-1.3
===============================================================================

To build kernel modules depending on OFED's modules, take the Modules.symvers
file from <prefix>/src/openib/Module.symvers (part of the kernel-ib-devel RPM)
and copy it to the modules subdir and then compile your module.

If <prefix>/src/openib/Module.symvers does not exist or it is empty, use the
create_Module.symvers.sh (a part of the ofed-docs RPM) script to create the
Module.symvers file.

See "Module versioning & Module.symvers" in the modules.txt from kernel
documentation (e.g. linux-2.6.20/Documentation/kbuild/modules.txt).


===============================================================================
5. Adding/Deleting a patch to OFED package
===============================================================================
If there is a need to add or delete a patch to OFED package one should use the 
ofed_patch.sh script, that is available under the docs directory.

This script supports kernel sources only.

Usage:
Add patch to OFED:
ofed_patch.sh  --add
               --ofed|-o <path_to_ofed>
               --patch|-p <path_to_patch>
               --type|-t <kernel|backport <kernel tag>|addons <kernel tag>>

Remove patch from OFED:
ofed_patch.sh  --remove
               --ofed|-o <path_to_ofed>
               --patch|-p <patch name>
               --type|-t <kernel|backport <kernel tag>|addons <kernel tag>>

Examples:
ofed_patch.sh --add --ofed /tmp/OFED-1.3/ --patch /tmp/cma_fix.patch --type kernel

ofed_patch.sh --remove --ofed /tmp/OFED-1.3/ --patch cma_fix.patch --type kernel

===============================================================================
6. Adding vendor specific actions to the installation of OFED
===============================================================================

Vendors that want to add actions to the install/uninstall process of OFED can
bind external scripts to hooks in install.pl and ofed_uninstall.sh.

4.1  Specifying vendor scripts and configuration parameters
-----------------------------------------------------------
This option is only available when installing ofed in non interactive mode.
Edit the OFED configuration file (ofed.conf ) and add the lines below (you don't have to use all of them).

# Script to run before install process starts
vendor_pre_install=my_pre_install.sh
# Script to run after install process finishes
vendor_post_install=my_post_install.sh
# Script to run before uninstall process starts
vendor_pre_uninstall=my_pre_uninstall.sh
# Script to run after uninstall process finishes
vendor_post_uninstall=my_post_uninstall.sh

You can also also add vendor specific configuration parameters. Lines that start
with vendor_config_ will not be parsed by install.pl and can be parsed by one
of the vendor scripts

vendor_config_something=value_for_that_something

Running ./install.pl -c ofed.conf in the OFED directory will now invoke the
relevant vendor specific actions.

4.2. Requirements from vendor scripts
-------------------------------------
The script files that are given to install.pl in ofed.conf should
	- be located in the root directory of OFED
	- return zero on success

If vendor script fails it fails the entire installation.

4.2 Skeleton for pre/post install vendor script
-----------------------------------------------
install.pl passes some useful installation variables to the vendor pre/post
install scripts environment. See the example below for a typical usage.

#!/bin/bash

eval $*

# The following env. parameters are set at this point
# 
# CONFIG: full path filename of the OFED configuration file
# RPMS: directory of binary RPMs
# SRPMS: directory of source RPMS
# PREFIX: prefix of installation
# TOPDIR: root of OFED package
# QUIET: quiet operation indicator

function readconf() {
        local config=$1
        while read line; do
            # skip comments
            [[ ${line:0:1} == "#" ]] && continue

            # skip empty lines
            [[ -z "$line" ]] && continue

            # parse line
            token=$(echo $line|cut -f1 -d=)
            value=$(echo $line|cut -f2 -d=)

        done < "$config"
}

readconf $CONFIG

exit 0




