SRP in OFED 1.0 Release Notes

June 2006


Overview
--------

	Note: This is a Beta release of the SCSI RDMA Protocol (SRP) Initiator.
	      It is intended for development use but not as a complete product.

The SRP standard describes the message format and protocol definitions required
for transferring commands and data between a SCSI initiator port and a SCSI
target port using RDMA communication service.


Software Dependencies
---------------------
The SRP Initiator depends on the installation of the OFED Distribution stack
with OpenSM running.

Major Features
--------------
This SRP Initiator is based on source taken from openib.org gen2 implementing
the SCSI RDMA Protocol-2 (SRP-2), Doc. no. T10/1524-D. See:
www.t10.org/ftp/t10/drafts/srp2/srp2r00a.pdf

The SRP Initiator supports:
- Basic SCSI Primary Commands -3 (SPC-3)
  (www.t10.org/ftp/t10/drafts/spc3/spc3r21b.pdf)
- Basic SCSI Block Commands -2 (SBC-2)
  (www.t10.org/ftp/t10/drafts/sbc2/sbc2r16.pdf)
- Basic functionalities, task management and limited error handling

Establishing an SRP Connection
------------------------------
The following steps describe how to load the SRP connection between the
Initiator and an SRP Target:
- Make sure that the ib_srp module is loaded, the SRP Initiator is connected
  to an SRP Target, and that an SM is running

- In order to create an SRP (SCSI) device under /dev  use the following format:

echo id_ext=[GUID value],ioc_guid=[GUID value],dgid=[port GID value],pkey=ffff,
service_id=[service[0] value] > /sys/class/infiniband_srp/srp-mthca[hca number]-
[port number]/add_target

- It is possible to find the needed target attributes for the former command
  by executing the ibsrpdm command. This command detects all the SRP Targets
  connected to the SRP Initiator. The ibsrpdm command can be found under
  /usr/local/ofed/sbin/ (or <prefix>/sbin/) and is part of the
  srptools RPM that can be installed as part of the OFED custom installation.
  Note that the execution of this command may take a while, and that the SM
  should be running.

- ibsrpdm Utility Output Example: 
>ibsrpdm
IO Unit Info:
    port LID:        0103
    port GID:        fe800000000000000002c90200402bd5
    change ID:       0002
    max controllers: 0x10

    controller[  1]
        GUID:      0002c90200402bd4
        vendor ID: 0002c9
        device ID: 005a44
        IO class : 0100
        ID:        LSI Storage Systems SRP Driver 200400a0b81146a1
        service entries: 1
            service[  0]: 200400a0b81146a1 / SRP.T10:200400A0B81146A1

- It is also possible to execute ibsrpdm with the -c option. Using this option
  outputs the target attributes in the same manner needed for the creation of
  an SRP device.  

- Output example for the same target:
> ibsrpdm -c
id_ext=200400A0B81146A1,ioc_guid=0002c90200402bd4,
dgid=fe800000000000000002c90200402bd5,pkey=ffff,service_id=200400a0b81146a1

- In order to detect all the SRP Targets connected to the SRP Initiator on a
  port other than the first port, use the following format:
  ibsrpdm -d /dev/infiniband/umad[port number-1]

- Using the output from the example above execute the following command
  to create an SRP device:
echo id_ext=200400A0B81146A1,ioc_guid=0002c90200402bd4,
dgid=fe800000000000000002c90200402bd5,pkey=ffff,service_id=200400a0b81146a1
> /sys/class/infiniband_srp/srp-mthca0-1/add_target
 
- The SRP connection should now be up.

- New SCSI devices can be listed using the 'fdisk -l' command. Moreover, a
  corresponding message with names of the new devices may be obtained from
  dmesg.

- When loading the ib_srp module, it is possible to set the module parameter 
  srp_sg_tablesize. This is the maximum number of gather/scatter entries per
  I/O (the default is 12).

- When creating an SRP device, it is possible to supply values for:
   > max_cmd_per_lun -  Default: 63
   > max_sect (short for max_sectors) - sets the request size of a command.
   > io_class  -  Default: 0x100 as in rev 16A of the specification.
		  Note: In rev 10 the default was 0xff00.
 

Known Issues
------------

- The SCSI device is sent offline when a link goes down for a long time, when
  the subnet manager goes down for a long time, or when a disk is removed from
  a target during run-time.

- A device going offline may cause a crash of the system. (One reason for this
  is the activation of scsidev.)

- No automatic discover/add of targets through device manager MAD queries (See
  above the section titled: "Establishing an SRP Connection".

- There is no notification mechanism to inform the initiator that a target has
  been removed.
