erf(5)

NAME
ERF (Extensible Record Format) file format

DESCRIPTION
Endace DAG Network Measurement Interface Cards produce trace files in their own native format, known as the Extensible Record Format (ERF). An ERF file consists of a series of records, each describing one packet.

An ERF file consists only of ERF records; there is no special file header. This allows concatenation and splitting to be performed arbitrarily on ERF record boundaries.

Generic Variable Length Record

With the exception of the timestamp, all fields are in network byte order. All payload data is captured as a byte stream; no byte reordering is applied.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |     (rlen - 16) bytes of payload     |
    +--------------------------------------+

timestamp (64 bits)
    The timestamp is a little-endian (x86 native byte order) 64-bit fixed point number representing the time since the Epoch (midnight on 1 January 1970). This timestamp is hardware generated when the packet arrives at the DAG card.

    The high 32-bits contain the number of seconds since the Epoch, while the lower 32-bits contain a binary fraction of one second. This allows an ultimate resolution of 2^{-32} seconds, or approximately 233 picoseconds. The actual resolution depends on the specific model of DAG card, with all unused lower bits in the timestamp being set to zero. In this way the interpretation of the timestamp does not need to change when higher resolution clock hardware is available.

    An advantage of the ERF timestamp format is that the difference between two timestamps can be found with a single 64-bit subtraction, without the need to check for overflow/underflow (as required when comparing structures of the kind used by gettimeofday(2) or nanosleep(2)).

    The following code sample demonstrates how a 64-bit ERF timestamp (erfts) can be converted into a struct timeval representation (tv):

        uint64_t lts;
        struct timeval tv;
        lts = erfts;
        tv.tv_sec = lts >> 32;
        lts = ((lts & 0xffffffffULL) * 1000 * 1000));
        lts += (lts & 0x80000000ULL) << 1; /* rounding */
        tv.tv_usec = lts >> 32;
        if (tv.tv_usec >= 1000000)
        {
            tv.tv_usec -= 1000000;
            tv.tv_sec += 1;
        }

type (8 bits)
    This field contains an enumeration of the frame subtype.

    0   TYPE_LEGACY

    1   TYPE_HDLC_POS: Packet-over-SONET (PoS) with HDLC framing.

    2   TYPE_ETH: Ethernet.

    3   TYPE_ATM: ATM cell.

    4   TYPE_AAL5: Reassembled AAL5 frame.

    5   TYPE_MC_HDLC: Multichannel HDLC frame.

    6   TYPE_MC_RAW: Multichannel raw link data.

    7   TYPE_MC_ATM: Multichannel ATM cell.

    8   TYPE_MC_RAW_CHANNEL: Multichannel raw link data.

    9   TYPE_MC_AAL5: Multichannel AAL5 frame.

    10  TYPE_COLOR_HDLC_POS: Packet-over-SONET with HDLC framing and packet classification information (from an IP Filtering module).

    11  TYPE_COLOR_ETH: Ethernet with packet classification information (from an IP Filtering module).

    12  TYPE_MC_AAL2: Multi-channel AAL2 frame.

    13  TYPE_IP_COUNTER:

    14  TYPE_TCP_FLOW_COUNTER:

    15  TYPE_DSM_COLOR_HDLC_POS: Packet-over-SONET with HDLC framing and packet classification information (from HMM Filtering firmware).

    16  TYPE_DSM_COLOR_ETH: Ethernet with packet classification information (from HMM Filtering firmware).

    17  TYPE_COLOR_MC_HDLC_POS: Multichannel HDLC frame with packet classification information (from an IP Filtering module).

    18  TYPE_AAL2: Reassembled AAL2 frame.

    32-47
        Reserved for CDK Users.

    48  TYPE_PAD: Pad record.

flags (8 bits)
    This field contains the capture interface identifier and a series of 1-bit flags as follows:

    1-0 Capture interface (0-3, represented as a 2-bit unsigned integer).

    2   Varying record lengths present.

    3   Truncated record (insufficient buffer space).

    4   Receive (RX) error (link-layer error).

    5   Internal (DS) error.

    6-7 Reserved for future use.

rlen (16 bits)
    Total length of the record transferred over the PCI bus to memory.

lctr (16 bits)
    A 16-bit counter that records the number of packets lost since the previous record. This counter starts at zero and sticks at 0xFFFF.

wlen (16 bits)
    The packet length, including some protocol overhead. The exact interpretation of this number depends on the physical link.

payload
    The contents of the payload bytes depends on the type of link being used for capture. The following sections provide more detail about the remainder of the packet for each of the various ERF types.

Type 1 PoS HDLC Variable Length Record

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |              HDLC header             |
    +--------------------------------------+
    |     (rlen - 16) bytes of payload     |
    +--------------------------------------+

HDLC header
    The HDLC header from the packet. This may vary (4 or 6 bytes) depending on the protocol.

Type 2 Ethernet Variable Length Record

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +--------+---------+-------------------+
    | offset |   pad   |    (rlen - 18)    |
    +--------+---------+                   |
    |           bytes of payload           |
    +--------------------------------------+

offset (8 bits)
    The number of bytes not captured from the start of the frame. Typically used to skip link layer headers when not required in order to save bandwidth and space. This feature is currently unimplemented and the contents of this field should be disregarded.

pad (8 bits)
    The pad byte is included so that the IP header is 32-bit aligned. Its contents should be disregarded.

Type 3 ATM Cell Record
The ATM header in this format does not include the Header Error Correction (HEC) byte.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |              ATM header              |
    +--------------------------------------+
    |           48 bytes of cell           |
    +--------------------------------------+

Type 4 Reassembled AAL5 Frame Record
The ATM header in this format does not include the Header Error Correction (HEC) byte.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |              ATM header              |
    +--------------------------------------+
    |     (rlen - 20) bytes of payload     |
    +--------------------------------------+

Type 5 Multi-channel HDLC Frame Record

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |         multichannel header          |
    +--------------------------------------+
    |              HDLC header             |
    +--------------------------------------+
    |     (rlen - 24) bytes of payload     |
    +--------------------------------------+

flags (8 bits)
    Differences from the generic ERF field: the capture interface field is always zero; fixed length mode is not supported; the RX error bit is set if any multichannel header error bit was set.

multichannel header (32 bits)
    This header is divided into several bit fields:

    0-9 Connection number (0-1023). The actual number of connections supported depends on the DAG card.

    10-23
        Reserved for future use.

    24  Frame checksum (FCS) error.

    25  Short record error (record was less than 6 bytes).

    26  Long record error (record was greater than 2047 bytes).

    27  Aborted frame error.

    28  Octet error. The closing flag was not octet aligned after bit unstuffing.

    29  Lost byte error. The internal datapath had an unrecoverable error.

    30  First record. This is the first record received since this connection was configured.

    31  Reserved for future use.

Type 6 Multi-channel Raw Time Slot Link Record

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |         multichannel header          |
    +--------------------------------------+
    |  (rlen - 20) bytes of raw link data  |
    +--------------------------------------+

flags (8 bits)
    Differences from the generic ERF field: the capture interface field is always zero; fixed length mode is not supported; the RX error bit is set if any multichannel header error bit was set.

multichannel header (32 bits)
    This header is divided into several bit fields:

    0-3 Physical interface (0-15).

    4-24
        Reserved for future use.

    25  Short record error (record was less than 6 bytes).

    26  Long record error (record was greater than 2047 bytes).

    27-28
        Reserved for future use.

    29  Lost byte error. The internal datapath had an unrecoverable error.

    30  First record. This is the first record received since this connection was configured.

    31  Reserved for future use.

raw link data

    T1  24 bytes, one byte for each of the 24 time slots.

    E1  31 bytes of data for time slots 0 to 31. Slot 16 (used for signaling) is not captured.

    Framed E1
        30 bytes of data for time slots 1 to 31. Slot 0 (used for framing) is not captured. Slot 16 (used for signaling) is not captured.

Type 7 Multi-channel ATM Cell Record
The ATM header in this format does not include the Header Error Correction (HEC) byte.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |         multichannel header          |
    +--------------------------------------+
    |             ATM header               |
    +--------------------------------------+
    |           48 bytes of cell           |
    +--------------------------------------+

flags (8 bits)
    Differences from the generic ERF field: the capture interface field is always zero; fixed length mode is not supported; the RX error bit is set if any multichannel header error bit was set.

multichannel header (32 bits)
    This header is divided into several bit fields:

    0-9 Connection number (0-1023). The actual number of connections supported depends on the DAG card.

    10-15
        Reserved for future use.

    16-19
        Physical port the cell was captured on (0-15).

    20-23
        Reserved for future use.

    24  Lost byte error. The internal datapath had an unrecoverable error.

    25  HEC corrected.

    26  OAM cell CRC-10 error (not implemented).

    27  OAM cell.

    28  First record. This is the first record received since this connection was configured.

    29-31
        Reserved for future use.

Type 8 Multi-channel Raw Link Data Record

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |         multichannel header          |
    +--------------------------------------+
    |  (rlen - 20) bytes of raw link data  |
    +--------------------------------------+

flags (8 bits)
    Differences from the generic ERF field: the capture interface field is always zero; fixed length mode is not supported; the RX error bit is set if any multichannel header error bit was set.

multichannel header (32 bits)
    This header is divided into several bit fields:

    0-9 Connection number (0-1023). The actual number of connections supported depends on the DAG card.

    10-28
        Reserved for future use.

    29  Lost byte error. The internal datapath had an unrecoverable error.

    30  First record. This is the first record received since this connection was configured.

    31  Reserved for future use.

Type 9 Multi-channel AAL5 Frame Record
The ATM header in this format does not include the Header Error Correction (HEC) byte.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |         multichannel header          |
    +--------------------------------------+
    |             ATM header               |
    +--------------------------------------+
    |   (rlen - 24) bytes of AAL5 frame    |
    +--------------------------------------+

flags (8 bits)
    Differences from the generic ERF field: the capture interface field is always zero; fixed length mode is not supported; the RX error bit is set if any multichannel header error bit was set.

multichannel header (32 bits)
    This header is divided into several bit fields:

    0-10
        Connection number (0-2047). The actual number of connections supported depends on the DAG card.

    11-15
        Reserved for future use.

    16-19
        Physical port the cell was captured on (0-15).

    20  CRC checked.

    21  CRC error.

    22  Length checked.

    23  Length error.

    24-27
        Reserved for future use.

    28  First record. This is the first record received since this connection was configured.

    29-31
        Reserved for future use.

Type 10 Colored PoS HDLC Record
This record is similar to the Type 1 PoS HDLC record, except that the lctr field is replaced with a color field.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |      color       |       wlen        |
    +------------------+-------------------+
    |     (rlen - 16) bytes of payload     |
    +--------------------------------------+

color (16 bits)
    This field is a hardware generated tag indicating the result of a filtering or classification operation:

    0   Set if the packet should have been sent to receive stream 0.

    1   Set if the packet should have been sent to receive stream 2.

    2-15
        A 14-bit unsigned integer (0-16383) that corresponds to the filter rule this packet matched.

Type 11 Colored Ethernet Record
This record is similar to the Type 2 Ethernet record, except that the lctr field is replaced with a color field.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |      color       |       wlen        |
    +--------+---------+-------------------+
    | offset |   pad   |    (rlen - 18)    |
    +--------+---------+                   |
    |           bytes of payload           |
    +--------------------------------------+

color (16 bits)
    This field is a hardware generated tag indicating the result of a filtering or classification operation:

    0   Set if the packet should have been sent to receive stream 0.

    1   Set if the packet should have been sent to receive stream 2.

    2-15
        A 14-bit unsigned integer (0-16383) that corresponds to the filter rule this packet matched.

Type 15 DSM Colored PoS HDLC Record
This record is similar to the Type 1 PoS HDLC record, except that the lctr field is replaced with a color field.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |      color       |       wlen        |
    +------------------+-------------------+
    |     (rlen - 16) bytes of payload     |
    +--------------------------------------+

color (16 bits)
    This field is a hardware generated tag indicating the result of a Data Stream Management (DSM) operation:

    0-5 The number of the receive stream that the packet should have been sent to. This number corresponds to the rx streams only, therefore 0 refers to rx stream 0, 1 refers to rx stream 2, 2 refers to rx stream 4 and so on.

    6-13
        Each bit refers to the output of a filter in the DSM module, bit 6 refers to filter 0, bit 7 refers to filter 1 and so on.

    14  Bit that indicates the output of the CRC hash load balancing algorithm for the packet record.

    15  Bit that indicates the output of the parity hash load balancing algorithm for the packet record.

Type 16 DSM Colored Ethernet Record
This record is similar to the Type 2 Ethernet record, except that the lctr field is replaced with a color field.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |      color       |       wlen        |
    +--------+---------+-------------------+
    | offset |   pad   |    (rlen - 18)    |
    +--------+---------+                   |
    |           bytes of payload           |
    +--------------------------------------+

color (16 bits)
    This field is a hardware generated tag indicating the result of a Data Stream Management (DSM) operation:

    0-5 The number of the receive stream that the packet should have been sent to. This number corresponds to the rx streams only, therefore 0 refers to rx stream 0, 1 refers to rx stream 2, 2 refers to rx stream 4 and so on.

    6-13
        Each bit refers to the output of a filter in the DSM module, bit 6 refers to filter 0, bit 7 refers to filter 1 and so on.

    14  Bit that indicates the output of the CRC hash load balancing algorithm for the packet record.

    15  Bit that indicates the output of the parity hash load balancing algorithm for the packet record.

Type 17 Colored Multi-channel PoS HDLC Record

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       color      |       wlen        |
    +------------------+-------------------+
    |         multichannel header          |
    +--------------------------------------+
    |              HDLC header             |
    +--------------------------------------+
    |     (rlen - 24) bytes of payload     |
    +--------------------------------------+

color (16 bits)
    This field is a hardware generated tag indicating the result of a filtering or classification operation:

    0   Set if the packet should have been sent to receive stream 0.

    1   Set if the packet should have been sent to receive stream 2.

    2-15
        A 14-bit unsigned integer (0-16383) that corresponds to the filter rule this packet matched.

flags (8 bits)
    Differences from the generic ERF field: the capture interface field is always zero; fixed length mode is not supported; the RX error bit is set if any multichannel header error bit was set.

multichannel header (32 bits)
    This header is divided into several bit fields:

    0-9 Connection number (0-1023). The actual number of connections supported depends on the DAG card.

    10-23
        Reserved for future use.

    24  Frame checksum (FCS) error.

    25  Short record error (record was less than 6 bytes).

    26  Long record error (record was greater than 2047 bytes).

    27  Aborted frame error.

    28  Octet error. The closing flag was not octet aligned after bit unstuffing.

    29  Lost byte error. The internal datapath had an unrecoverable error.

    30  First record. This is the first record received since this connection was configured.

    31  Reserved for future use.

Type 18 Reassembled AAL2 Frame Record
The ATM header in this format does not include the Header Error Correction (HEC) byte.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +------------------+-------------------+
    |            AAL2 ext header           |
    +--------------------------------------+
    |             ATM header               |
    +--------------------------------------+
    |   (rlen - 24) bytes of AAL2 frame    |
    +--------------------------------------+

AAL2 ext header (32 bits)
    This header is divided into several bit fields:

    0-7 Channel Identification Number (cid)

    8-15
        MAAL Error Number as specified in ITU I.363.2. This field is only valid if the MAAL Error Indication bit is set.

    16  MAAL Error Indication, will be set if the frame has a MAAL error otherwise it is cleared.

    17-31
        Reserved for future use.

Type 48 Pad Record
This record is used to pad or flush records out of a DAG card. It contains no useful information and should be discarded.

     BYTE 3    BYTE 2      BYTE 1    BYTE 0
    +--------------------------------------+
    |               timestamp              |
    +--------------------------------------+
    |               timestamp              |
    +--------+---------+-------------------+
    |  type  |  flags  |       rlen        |
    +--------+---------+-------------------+
    |       lctr       |       wlen        |
    +--------------------------------------+
    |   (rlen - 16) bytes of padding       |
    +--------------------------------------+

timestamp (64 bits)
    The timestamp is all zeros.

flags (8 bits)
    The flags field is always 0.

wlen (16 bits)
    The wire length field is always 0.

lctr (16 bits)
    The loss counter field is always 0.

SEE ALSO
dagaal5(1), dagbits(1), dagbtx(1), dagbug(1), dagcam(1), dagchan(1), dagclock(1), dagconvert(1), dagflood(1), dagfour(1), dagfwddemo(1), daggen(1), daginf(1), dagld(1), dagpps(1), dagreset(1), dagrom(1), dagsar(1), dagsix(1), dagsnap(1), dagthree(1), dagtime(1), dagvcam(1), dagwatchdog(1), gettimeofday(2), nanosleep(2), pbmbug(1).

LAST UPDATED
$Author: ben $
$Revision: 7183 $
$Date: 2007-08-02 11:29:09 +1200 (Thu, 02 Aug 2007) $

COPYRIGHT
(c) 2004-2005 Endace Technology, Hamilton, New Zealand. All Rights Reserved.