#
# Copyright (C) 2001 by USC/ISI
# All rights reserved.
#
# Redistribution and use in source and binary forms are permitted
# provided that the above copyright notice and this paragraph are
# duplicated in all such forms and that any documentation, advertising
# materials, and other materials related to such distribution and use
# acknowledge that the software was developed by the University of
# Southern California, Information Sciences Institute.  The name of the
# University may not be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
# WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
#
# This text is a simple document that serves to explain how to use 
# SAMAN ModelGen
#
# This work is supported by DARPA through SAMAN Project
# (http://www.isi.edu/saman/), administered by the Space and Naval 
# Warfare System Center San Diego under Contract No. N66001-00-C-8066
#

                  How to use ModelGen (Model Generator)
	         --------------------------------------

Kun-chan Lan                 Version 1.0                               10/9/01

1. Introduction

This is a quick description of how to use Model Generator (ModelGen) to
rapidly generate traffic model from raw trace for simulation and 
traffic analysis.  ModelGen contains a set of C/Perl/Tcl programs. It 
takes a tcpdump file as input and outputs a set of CDF 
(Cumulative Distribution Function) files that model the user-level
statistics of targeted traffic.  Currently it only supports Web traffic.

2. Usage

First, type "make all" under Unix command line 

    to make "http_connect" and "http_active"

then type
    ModelGen <tcpdump file> <threshold for user think time> <netwrok prefix>

It needs two arguments. First argument is the file name of tcpdump
trace (generated by tcpdump -w option). The second argument is the
threshold time value (in millisecond) that distinguishes idle periods
in order to infer user "think" times between requests for new top-level
pages. The last argument is the network prefix used to distinguish
inbound traffic from outbound traffic.

3. List of Files

ModelGen contains a set of programs, including
(1) http_connect
(2) http_active
(3) delay.tcl
(4) outputCDF
(5) time-series.pl

The files http_connect and http_active are 2 C programs  
written and contributed by Don Smith <smithfd@cs.unc.edu> and
Felix Hernandez Campos <fhernand@cs.unc.edu> from The University
of North Carolina at Chapel Hill. These two programs were also used
for their SIGMETRICS 2001 paper "What TCP/IP Protocol Headers Can Tell 
Us About the Web". The program http_connect is used to performs an analysis
of tcpdump output and produce a summary of the TCP connections used for
HTTP. It assumes that tcpdump has been filtered for packets that are
from TCP source port 80 and the result has been sorted so that packets 
are in ascending time order within each TCP connection. The program 
http_active is used to create an activity trace (summary form) of web
browsing clients with respect to three types of activities: client
sending request data, server sending responds data, client is idle.
For more detailed description of these two programs, please see
accompanying files trace_processing.pdf and output_format.pdf (also
contributed by UNC)

The file delay.tcl is a Tcl script that estimates the delay between
each Web client and server pair. It takes a tcpdump trace as input
and assumes that trace has been filtered for packets that only originates
from or desinate to port 80. It looks at the beginning of each TCP
connection and see how much time it takes for the SYN/SYN-ACK handshake
to estimate the Round Trip time (RTT) delay for each
connection. It finally outputs a summary form of averaged RTT for
each source/destination pair (which might contain several connections
within the entire trace)

The file outputCDF is a Perl script that takes the output of
http_active to infer a set of user-level statistics, including
(a) user session inter-arrival time
(b) number of pages per user session
(c) page inter-arrival time
(d) page size
(e) object inter-arrival time
(f) object size
(g) client request size
(h) ratio between persistent and non-persistent connection
(i) server popularity
The output will be a set of CDF files that describe the above statistics
in 3-column format (take pagesize.dat.cdf for example)
1st column: page size (in KB)
2nd column: accumulated number of samples
3rd column: accumulated probability

The file time-series.pl is a Perl script that takes the output
of tcpdump trace (that ASCII print lines that tcpdump generates to
stdout) and produce a time series of traffic volume (the unit block is 
1 millisecond) for use in further scaling analysis of traffic.

Here is a summary of the order of execution for each program in ModelGen:
(1) filter tcpdump trace for TCP port 80

    ./tcpdump -n -tt -r $1 tcp port 80 > $1.www
    ./tcpdump -n -tt -r $1 tcp src port 80 > $1.http-srv

(2) run http_connect 

    sort -s -o $1.http-srv-sort +1 -2 +3 -4 +0 -1 -T /tmp $1.http-srv
    http_connect -r $1.http-srv -w $1.http-srv.connect
    grep "ACT" $1.http-srv.connect > $1.http-srv.connect.time
    sort $1.http-srv.connect.time > $1.http-srv.connect.time.sort

    INPUT: (1) in ASCII format
    OUTPUT: summary of http connections
(3) run http_active 

    sort -s -o $1.http-srv.connect.sort +1 -2 +0 -1 -T /tmp $1.http-srv.connect
    http_active -r $1.http-srv.connect.sort -w $1.http-srv.active -I $2

    INPUT: output of (2) 
    OUTPUT: summary of http client request and server response
(4) compute delay based on each SYN/SYN-ACK pair

    ./delay.tcl $1.www > $1.sync
    sort -s -o $1.sync.sorted -T /tmp $1.sync
    pair.tcl $1.sync.sorted >  $1.sync.delay
    sort -s -o $1.sync.delay.sorted -T /tmp $1.sync.delay
    awk -f delay.awk < $1.sync.delay.sorted > delay.dat

    INPUT: (1) in ASCII format
    OUTPUF: CDF that characterize the averaged RTT for each 
    source/destination pair
(5) compute and output CDF files that model Web traffic

    outputCDF $1.http-srv.active.activity
  
    INPUT: output of (3) 
    OUTPUT: CDF files of 
    (a) session inter-arrival 
    (b) number of pages per session 
    (c) page inter-arrival 
    (d) page size 
    (e) object inter-arrival 
    (f) object size 
    (g) request size 
    (h) ratio between persistent and non-persistent connection
    (i) server popularity  
(4) compute time series in 1ms block

    bw.tcl $1.http-srv
    cat $1.http-srv.bw | time-series.pl > $1.time-series

    INPUT: (1) in ASCII format
    OUTPUT: a time series of traffic size in 1ms block                                                                
4. NS scripts

The file isiweb.tcl is a ns script that demonstrates how to use
the output of ModelGen (the set of CDF files) to simulate the
traffic collected at ISI/USC gateway link. To execute it, 
assuming you already have ns (snapshot after 10/10/01) installed 
in your system, type "ns isiweb.tcl" from the command line. It 
takes ~1 hour on a Red Hat Linux 7.0 Pentium II Xeon 450 MHz PC 
with 1GB physical memory. The complete example including the 
CDF files that model ISI traffic can be found under 
<your ns root>/tcl/ex/empweb/ of your ns distribution



Contact person

    Kun-chan Lan, USC/ISI
    email: kclan@isi.edu
    phone: 310-448-8260

