README-linux

README for linux distribution

The directory
	scripts
contains some sample initialization scripts that
can be modified for your system to automatically
load the gm driver and start the mapper.
We have not used these scripts at Myricom. They
were supplied by a customer.


WARNING: When building/linking GM applications, you must do so
         on a linux box that matches the OS version of the machine
         you will be running on. You cannot compile on a 2.0.x
         machine and run the executable on a 2.2.x machine.

All the linux drivers support IP access to GM through a normal
"myri0" device.


Compiling from source
--------------------
If you compile GM yourself from source for linux-2.2.x, you need to compile
the code on a machine that is running the same flavor of kernel
that you expect to install the driver on. If your cluster is
runnning SMP linux, you need to compile GM on a machine running
SMP linux. If your cluster is UP, then you need to compile on
a machine that is running UP linux.

There are some additional configure options for linux
 --with-linux=<dir>          point to a different linux-kernsrc
 --enable-linux-smp          force -D__SMP__ to be defined 
 --enable-linux-modversions  force -DGM_NEED_MODVERSIONS to be defined
                             you also need to be sure your linux-kernsrc tree has
                             modules enabled

If you get symbol errors when loading the GM module, it is most likely
because the kernel has versioned symbols and the module does not, or
vice versa. You can cat /proc/ksyms to see what the kernel symbols
look like. 

Here's what you need to do to get your kernel source
tree properly configured.
Note that GM runs under any kernel, the exact version
numbers below are just an example.


1) uname -a

e.g.
frisbee.myri.com 413% rsh empire uname -a
Linux empire.myri.com 2.2.5-15smp #1 SMP Mon Apr 19 22:43:28 EDT 1999 i686 unknown

Notice that the kernel name is
	 2.2.5-15smp

Now edit /usr/src/linux/Makefile

frisbee.myri.com 415% head /usr/src/linux/Makefile 
VERSION = 2
PATCHLEVEL = 2
SUBLEVEL = 5
EXTRAVERSION = -10

In our case, you'd want to modify the makefile lines above to read

VERSION = 2
PATCHLEVEL = 2
SUBLEVEL = 5
EXTRAVERSION = -15smp


Then
cd /usr/src/linux
make xconfig
and turn on the parameters for module versioning [and SMP if your kernel
version in (1) was smp.

then

/bin/rm -f /usr/src/linux/include/linux/modules/*
make dep ; make clean ; make modules

Now the linux source tree is properly configured.

Now you can go back and untar the GM source and 
build it.

WARNING - Redhat ships config files
/usr/include/linux/autoconf-smp.h and autoconf-BOOT.h
that include the line
#undef CONFIG_BIGMEM
We're not sure why they do this, but it 
will make the GM driver not work properly
if you are running a BIGMEM-enabled kernel.
If so, then you need to comment out the #undef
in those files before compiling GM. Otherwise,
you'll see something like this

kernel: WARNING: drivers/linux/gm/gm_arch.c:2605:gm_linux_mmap():kernel:
kernel: Bad physical address in gm_linux_mmap
kernel:     phys=0x3ee40000  base=0xfb000000  span=0x1000000 sum=0xfc000000
kernel:     HIGH_THRESH = 0x3c000000  HIGH_MEM = 0x3c000000 PAGE_ZERO = 0x0


WARNING VA linux might be shipping mismatched ver files and kernel
binaries.

This note is from a customer with a VA linux software install.
#
#We figured out the problem.  The configuration that we were trying to 
#generate the .ver file from was not the same as the actual kernel 
#configuration.  So the wrong symbol was being generated by the `make 
#dep`.
#
#Since we are using VA Linux Red Hat, there was actually a very simple 
#fix.  Chand directory to /usr/src/linux and execute `make mrproper; 
#make restore`.  This completely cleaned out the source directory and 
#then recreates the configuration based upon the current kernel. 
#After that a `make dep` generate the correct .ver files and a `make 
#modules` creates the correct modules for the running kernel.
#


Using Compaq Compilers for Alpha Linux (ccc cxx)
------------------------------------------------
setenv CC ccc
setenv CXX cxx
setenv CXXFLAGS \
    "-g -O2 -inline speed -x cxx -noexceptions -nocxxstd -using_std -w2"
setenv CFLAGS -gcc_messages
setenv KCC gcc
rm -f config.cache
./configure [your options]



x86 platforms
-------------
linux-2.0
---------
GM will compile to a module that loads on
both SMP and UP.

linux-2.2
---------
GM will compile a module that will run on either
an SMP or a UP system depending on what system it was
compiled on. We look at the symbols in /proc/ksym to
see if there are any symbols with smp in the name.

alpha platforms
---------------
GM will compile to a driver that runs on any alpha platform.
It dynamically detects the type of system you are on and
adjusts internal configuration accordingly. 

ultrasparc platforms
--------------------
GM will only compile and run under sparc64 linux (Ultrasparc)

powerPC platforms
-----------------
GM has been tested on G3 and G4 platforms using Yellow Dog and Black Lab Linux.


Improving IP performance
------------------------
For linux you want to crank open the tcp windows.


echo "262144" > /proc/sys/net/core/rmem_max
echo "262144" > /proc/sys/net/core/wmem_max
echo "262144" > /proc/sys/net/core/wmem_default
echo "262144" > /proc/sys/net/core/rmem_default

In the "old" days you also needed to do this.
We're not sure if linux-2.4 still needs it.
in linux/include/net/tcp.h

   replace

#define MAX_WINDOW    32767

with the value of your choice (200k~500k might be good)

check /proc/sys/net/ipv4/tcp_window_scaling is enabled (it should be
by default)

Play with the buffer sizes of netperf or your favorite net tester.

You definitely want to use NFS v3 over TCP to get good NFS bandwidth.
Otherwise, you are latency dominated and myrinet IP and  ethernet
IP will be about the same.

