dagapi(3)

NAME
DAG API - capture and transmit network traffic on Endace DAG cards

SYNOPSIS
#include "dagapi.h "

DESCRIPTION

Endace DAG cards are a series of high performance PCI (including PCI-X and PCIe) cards designed for packet capture on ATM cell, PoS and Ethernet networks.

Packet and cell transmission is supported on some cards. Various versions have been produced supporting different physical layer interfaces. There are methods for accessing the DAG cards for both passive packet capture and for packet transmission.

Endace cards currently support the following line rates and networks:

1.544/2.048 Mbps
    T1/E1

10 Mbps
    Ethernet

100 Mbps
    Fast Ethernet

155 Mbps
    OC-3/STM-1

622 Mbps
    OC-12c/STM-4

1000 Mbps
    Gigabit Ethernet

2488 Mbps
    OC-48/STM-16c

9953 Mbps
    10 Gigabit Ethernet (LAN)

10312 Mbps
    OC-192c/STM-64c, 10 Gigabit Ethernet (WAN)

Collecting Network Traffic

In passive capture mode all DAG cards reflect the arrival of a packet, or ATM cell, as a record within a large circular memory buffer provided by the PC host computer system. An Extensible Record Format (erf(5)) record consists of a fixed size header, which includes a high precision time stamp, followed by packet data. This is a portion, or all, of the original packet present on the network link under observation followed by a small number of optional padding bytes to enforce alignment to a 32- or 64-bit boundary. There are three main ways to use DAG cards to capture packets.

The first method is to use the dagsnap(1) application as a simple way to capture network data. dagsnap(1) can write a file to disk containing the ERF records in sequential order as presented by the DAG card. Such a trace file may later be processed by an analysis package. If no output filename is specified, dagsnap(1) will write to standard output, allowing the data to be piped into an analysis package that reads from standard input. Although simple to prototype, this is not a high performance interface since all network trace data must pass through a UNIX pipe causing multiple memory copies and creating a bottleneck in the CPU-memory path. PC-based computer systems can achieve between 400 and 1000 mebibytes/sec read/write performance when accessing non-cached data in main memory. When data is copied twice (or more) the effective throughput lowers to a bandwidth comparable to Gigabit Ethernet. DAG cards can currently deliver up to 6.4 Gigabits per second peak data rate from a single DAG 6.x OC-192c card to main memory.

The libpcap library available from www.tcpdump.org now includes direct support for DAG cards (see pcap(3)), which allows an application written for the libpcap API to capture directly from DAG cards with no source code changes (re-linking against a DAG-enabled libpcap may be necessary). A DAG card is not a NIC (network interface card), and much more efficient memory-mapped access is provided than is possible using NIC cards and drivers. This allows libpcap applications zero-copy access to packet headers and contents, but reduces the timestamp resolution as libpcap uses a struct timeval (seconds and microseconds) for recording packet arrival time.

The third and most efficient method for retrieving network data is the native C language API. It provides the highest performance by providing a low-overhead zero-copy memory-mapped interface to the DAG card.

DAG cards consist of:

1   Line interface hardware.

2   A physical layer (either an ASIC or implemented in an FPGA).

3   A packet processor that timestamps packet arrivals using the DAG Universal Clock Kit (DUCK) and creates ERF records.

4   An optional processing element.

5   The PCI Burst Manager (PBM) which writes ERF records over the PCI bus into the host computer.

The physical layer interface is configured with an external tool, e.g. dagthree(1) for the DAG 3.x series cards. These tools report and set physical layer attributes such as SONET scrambling and loopback. They are designed to be easily scriptable, and should be called at least once before the measurement software is run. The packet processor controls such things as the ERF record length and padding options. These packet processor options are also configurable with external tools.

The PCI Burst Manager

Configuration of the PCI Burst Manager is performed with the dagfour(1), dagsix(1), and dagthree(1) tools, and is limited to the amount of memory allocated to each stream. Physical memory is reserved per card at boot time by the dagmem driver, and this memory can then be divided into one or more large circular buffers, known as streams. Receive streams use even identifiers (0, 2, 4, ...) and transmit streams use odd identifiers (1, 3, 5, ...). A particular firmware image set for a DAG card will have a maximum number of transmit and receive streams that it can support. The traditional receive-only DAG firmware image supplied with all DAG cards supports a single receive stream (i.e. stream 0), and no transmit streams. Firmware that supports other stream configurations may be available under separate licenses.

During receive operations the PCI Burst Manager receives ERF records from the packet processor, and writes them into a receive stream buffer in host PC memory via Bus Mastering DMA. The PCI Burst Manager indicates its position in the buffer to the DAG API via a producer pointer, and will cease writing when the buffer is full. When this happens ERF records are lost until the consumer frees some space in the buffer. Lost packets are counted and reported both in-band (via the lctr field in the ERF record) and out-of-band (via so-called 'safety net' messages written by the driver to the system log and console). The DAG API manages the corresponding consumer pointer, moving it forward as ERF records are consumed by user applications in order to clear more space in the circular buffer.

During transmit operations the user application writes ERF records to be transmitted into a transmit stream buffer, which is also a large circular buffer in PC memory. The buffer pointers are then updated to reflect the new data available for transmission. The PCI Burst Manager reads the data from the stream via Bus Mastering DMA and sends it to the physical layer for transmission. ERF records must be 64-bit aligned for transmission.

The C Language DAG Capture API

The programmer's interface to the DAG cards is exposed via a static or shared C library, libdag. Although it is possible to make operating system calls directly to the DAG device driver (dag(4)), this is discouraged as the kernel driver interface is subject to change without notice. The use of this library also aids portability. The current version of the API is intended to provide data capture and transmission functionality. Functions include:

1   Card open.

2   Buffer memory mapping.

3   Buffer pointer handling.

4   Packet record reception.

5   Packet record transmission.

6   Card close.

Functions provided in the separate Configuration and Status API include:

1   Card reset.

2   Xilinx image loading.

3   Hardware configuration.

4   Status and statistics information.

The Configuration and Status API functionality can also be accessed through the utilities provided with the standard software release: dagclock(1), dagfour(1), dagld(1), dagreset(1), dagrom(1), dagsix(1), dagthree(1). The public interface to libdag is defined in the header file dagapi.h.

The Receive Programming Model

API v1.1 and API v1.2 supported DAG cards operating in a passive receive mode only. This capability was represented as a single receive only data buffer.

API v1.3 introduced support for packet transmission, and the concept of multiple buffers per DAG card, called streams. Each stream consists of a buffer which can be used for either receiving or transmitting ERF records. Each stream can be addressed individually, with each receive stream presenting similar functionality to the traditional single buffer API. Capture on each stream can be started and stopped independently, but the streams cannot be considered fully independent as some operations (such as resets) are available only globally and thus affect every stream on a card.

There are two access methods available for receive streams. The dag_advance_stream(3) routine operates similarly to the dag_offset(3) routine from API v1.2. It provides a pair of record aligned pointers into the stream buffer. The bottom pointer indicates where the next ERF record begins. The top pointer indicates the upper limit of available data. The user application steps through the records using the record length field from the ERF header until it reaches the top pointer. The top pointer is then passed in as the as the new bottom pointer, freeing the processed space in the buffer. When dag_advance_stream(3) returns it will indicate the boundaries of a new block of data. dag_advance_stream(3) is the most efficient interface, as it can pass large blocks of records to the user with a single call. However, it does require the user to parse through the buffer content in order to process each record.

A simpler interface is provided - dag_rx_stream_next_record(3) can be used in place of the dag_advance_stream(3). This routine is called repeatedly, each time returning a pointer to the beginning of the next available ERF record. Buffer pointer management, freeing buffer space, and stepping through the available records are all implemented by the library. The disadvantage is that a function call for every ERF record is less efficient than the block oriented dag_advance_stream(3) routine, with approximately 10% higher overhead.

V1.4 of the DAG API introduced only small changes from v1.3, primarily aimed at making the API more consistent by changing error return codes. The sole new routine is dag_get_stream_buffer_level(3), which can be used to report receive or transmit stream status.

V1.6 of the DAG API introduced dag_rx_stream_next_inline(3), a routine for zero-copy inline forwarding of packets on transmit-capable DAG cards.

The Transmit Programming Model

The DAG API v1.3 introduced the ability to transmit packets, with appropriate firmware. Packet transmission is accomplished via transmit streams, which can be thought of as receive streams operating in reverse. Two access methods are provided for packet transmission, dag_tx_stream_commit_bytes(3) and dag_tx_stream_commit_bytes(3). ERF records must be 64-bit aligned for transmission.

If the user application can construct its packet in a buffer provided by the DAG API, the user can call dag_tx_get_stream_space(3) to get a pointer into the stream buffer at which it can directly write one or more ERF records. The user then calls dag_tx_stream_commit_bytes(3) to commit the appropriate number of bytes for transmission. This process is repeated to send further packets.

If the user wishes to transmit ERF records that are already present in memory, then dag_tx_stream_copy_bytes(3) should be used. This routine copies the ERF records from the user buffer into the DAG stream buffer and commits them for transmission. It is not necessary to call dag_tx_get_stream_space(3). dag_tx_stream_copy_bytes(3) is less efficient as the ERF records must be copied at least once by the CPU from the user buffer to the stream buffer.

FILES
dagapi.h

SEE ALSO
dag(4)dagclock(1), dagconfigapi(3), dagfour(1), dagld(1), dagreset(1), dagrom(1), dagsix(1), dagsnap(1), dagthree(1), dag_advance_stream(3), dag_get_stream_buffer_level(3),dag_offset(3), dag_rx_stream_next_inline(3),dag_rx_stream_next_record(3), erf(5), pcap(3).

BUGS
Report bugs to <support@endace.com>.

LAST UPDATED
$Author: koryn $
$Revision: 2474 $
$Date: 2005-11-10 11:13:19 +1300 (Thu, 10 Nov 2005) $

COPYRIGHT
(c) 2004-2005 Endace Technology, Hamilton, New Zealand. All Rights Reserved.