README for shak
See file INSTALL for installation instructions.

----------------------------------------------------------------------

MAKING SHAK
        Do:
            zcat <shak-2.2a.tar.gz | tar xvf -
            cd shak
            make

UPGRADING OR RE-INSTALLING SHAK from a previous
standard "make install"
        Do:
	    make uninstall && make install

        this is configuration file safe.

-----------------------------------------------------------------------

The AUTOMATIC INSTALLATION is recommended for
evaluation purposes and offers easy path to getting
the examples below working 'out of the box'.  It
allows installation in a temporary directory and 
automatically configures itself to work in that place.

See file INSTALL for installation instructions.

------------------------------------------------------------------------
WHAT IS SHAK ?

      Shak is program that runs your backup program.  Shak
      offers a way to write a single backup program that works on 
      multiple hosts and for multiple backup types.  To do
      this you must write your backup program in terms of the
      shak environment variables, when shak runs shak sets up 
      these environment variables and runs your program.

      Shak keeps track of all your backups by maintaining a tape index file
      for each tape cartridge.  When shak runs a backup, it reads all
      the index files and sets a variable containing the date of the last
      run backup on that host among others, your program may use this 
      to make an incremental or differential backup. 
      The Tape Index File defines the contents of the tape allowing you
      to append backups to a tape making a record of the tape contents.  It
      even stores the tape index file on the tape so it can be recovered.

      Shak also can monitor the stderr stream of your user backup and sends
      notification (mail) on errors.  This feature can be used to obtain the
      exit status of a remote backup.  This feature also provides a signal
      interface which allows a clean shutdown of all processes.

      Shak makes extensive use of mt(1), hence, if your tape drive works 
      well with mt(1) it will work well with shak.
      Since shak does not impose a backup program, shak can't do much more
      than described above.

----------------------------------------------------------------------
TESTING SHAK ON YOUR SYSTEM

**  See the file ./INSTALL,  TESTING SHAK Section. **
	
	SYSTEMS THAT *MAY* HAVE PROBLEMS:
		A Intel RedHat 4.1 System demonstrated
		a problem running awk(1). (shak0_qfilter
		misbehaved for no apparent reason, same input
		and binary on other system did OK).
	
	SYSTEMS THAT PASSED:
		Intel Redhat 4.0  May fail with /lib/libc.so.5.3.12
		                  Upgrade shared libs to
				  to fix problem with sed(1).
				  Passes with /lib/libc.so.5.4.7

		CND 1.0
		CND 1.0, Upgraded to RHS 3.0.3
		Intel RedHat 4.2
		Intel OpenLinux 1.0

		(no others tested)

-----------------------------------------------------------------------
SETTING UP REMOTE SHELL COMMAND

Shak requires that rsh(1) for the users running shak be configured
correctly.  This most often involves editting the ~/.rhosts file in
each users home directory.  It should contain at least the hostnames 
"localhost" and "whatever-your-hostname-is".
If your hostname is "kermit", the .rhosts file should look like:

localhost
kermit
kermit.<your-domain-name>

To verify operation, the following should work:

rsh localhost date
rsh kermit date

------------------------------------------------------------------------
To Read the man page:

    nroff -Tascii -mandoc <proper/man/shak.8 | less
   and
    nroff -Tascii -mandoc <proper/man/shak.5 | less

----------------------------------------------------------------------
THE EXAMPLE BACKUP PROGRAM: user/script/backupshak.sh

      The example backup script makes a backup of the shak installation
      on the local host.

      You can easily modify it to backup the files you want by editting
      the file.  This makes you a shell programmer.

      Other backup programs are:
          backuproot.sh
	  		Backs up everything except /home and /usr/local

          backuplocal.sh
			Backs up /home and /usr/local

       * The $Id$ string is important to shak, It identifies
         the backup program.  If you don't want to use GNU RCS you can
	 edit the string directly, shak won't know the difference but
	 this is not recommended.  See the man page, shak.8, for more
	 information on the user backup program and the design constraints
	 imposed by shak.


	* To make a full backup on tape labeled 1 and capacity of 
	  250000 blocks (capacity is not currently used by shak but
	  it is required when overwriting a tape.) This command will
	  not ask for any prompts or confirmation.
	     
	     /usr/sbin/shak -i 1 -x -c 250000 run -t 0 backuproot.sh

	  To append to the same tape.

	     /usr/sbin/shak -i 1 run -t 0 backuproot.sh

	  Incremental backups can be appended to the same tape:
	     
	     /usr/sbin/shak -i 1 run -t 3 backuproot.sh


----------------------------------------------------------------------
CONTENTS: (incomplete list)

shak/proper
	The main part of shak

shak/user/scpipt
	example network transparent backup scripts 
	that use the shak environment variables.

shak/user/src
	date parsing filter program.

shak/unsupported
	programs that have some usefullness
	in the user backup scripts.

----------------------------------------------------------------------
CONFIGURATION:
	DEFAULT FILE LOCATIONS UNLESS YOU EDIT THE SOURCE CODE:
	   directory: /etc/shak/
	   directory: /usr/etc/shak/
	    filename: shak.conf

	* create and edit /etc/shak/shak.conf  and/or
	         /usr/etc/shak/shak.conf
          
	  /etc/shak : sets local config items or
	  		all the config items
	  
	  /usr/etc/shak: (optional) sets items that
	                 are site-wide constants.

			
	* shak.conf looks something like
	  this:

domain:domainname
shakpath:/usr/local/lib/shak
loghost:localhost
indexhost:localhost
indexpath:/var/adm/backup/shak/index
tapehost:localhost
device:/dev/nst0
userpath:/usr/local/lib/backup/shak
logbasename:LOG_
logpath:/var/log/backup
remsh:rsh
useheaders:yes
mt_eom:eod
devicesecure:no
sitesecure:no
tapesecure:no


        to read shak.5 containing the definitions, type:
	nroff -Tascii -mandoc proper/man/shak.5 | less

	See ./INSTALL for more configuration requirements.

-------------------------------------------------------------------
MISC NOTES:

*  WATCHDOG TIMER: shak uses a timer to monitor most remote 
   shell operations.  If execution of the task exceeds the timeout
   value the command is killed and an error exit status is returned.
   The defaults are set in proper/bin/shak by the delay__* variables.

   The actual timer values are set by the delay_* variables in
   proper/bin/shak.  These variables can be set by the parent shell of shak
   when run, this will override the default.  Setting a variable to "0"
   will disable the timer for that operation category.
   The operation categories and variables and default values (sec) are :

      Time to perform any mt(1) tape operation : delay_tape   	6000

      Time to read a disk file: 	delay_readfile		120

      Time to read a tape header file:	 delay_readtape		120

      Time to write a header to tape:	delay_writeheader	120

      Time to run the user backup:	delay_userbackup	0

   The file proper/lib/shak_sick executes when the timer expires.  It
   takes the tape unit offline and writes to the log file thru logger(1)
   and sends mail to root.

*  The Makefiles are all written for Linux. Users
   of other platforms may have to edit the Makefiles.

*  Read the man pages, proper/man/shak.*, for documentation.

*  The 'fix' subcommand is the least tested of all
   the parts of shak.

*  Headers in this version are not compatible with version 0.96

-----------------------------------------------------------------------
USAGE EXAMPLES:

      * RUN THE TRIVIAL BACKUP, script/trivial.sh
      If trivial.sh is in SHAK_USERPATH on localhost.
  
          shak -q run trivial.sh
       
        This prints the environment interface.
        
	Now use the -D switch to change the tapedevice
	to /dev/nst0 on host H001.

          shak -q -D H001:/dev/nst0 run trivial.sh

      * Run a full backup that does not prompt the user.
        Using a 150Mb tape labeled 33.

          shak -m -i33 -c 150000 run -x -t full backupshak.sh

      * Run shak using /tmp/cfgfile as the only source of configuration
       information ( example /tmp/cfgfile must exist ).
          
	  shak -qdllg /tmp/cfgfile run trivial.sh

      *  MAKE A BACKUP OF THE SHAK DISTRIBUTION 
         run shak to backup shak. (may have to be root)
	   
	  shak run -t full backupshak.sh
      
         or to reduce tape manipulation, specify a tape number and
         and a tape size  and run with minimal checks.

          shak -mi16 -c 250000 run -t full backupshak.sh

         or specify expert mode and the tape size.
       
          shak -xmi16 -c250000  run -t full backupshak.sh

        This will run wihout any user prompts and overwrite
	the tape.

        These examples assumes that backupshak.sh is found
	in SHAK_USERPATH and all programs run by this
	example can be found. 
           
        backupshak.sh is an example and it makes a backup of 
        the ${SHAK_SHAKPATH} directory.
  
  If the index files and logs are written to ${SHAK_SHAKPATH}/var
  then the incremental and differential backups will include only these
  newly created files otherwise there won't be any files in SHAK_SHAKPATH
  that are new since the last backup.

 	* Now run incremental and differential types:
	      
	      shak run -t diff backupshak.sh
	      shak run -t incrs backupshak.sh

  Now look at the tape index file in SHAK_INDEXPATH and
  the log files in SHAK_LOGPATH to see what these backups did.

  MORE EXAMPLES:
 
       * make a full backup on tape 1, without any prompts  or overwrite
         warnings and with minimal tape checks.
	  
	  shak -xmi1 -c250000 run -t0 backupshak.sh

       * run shak with a environment variable specified on the command line
         of a bourne-like shell.
	 The '-E' option must be used or the variable is ignored.

	  SHAK_TAPEDEVICE=/dev/null  shak -Eq run trivial.sh

           This is usefull if you are running queries (modes of shak
	   that don't reqire a loaded tape) and want to
	   make sure nothing gets written to the tape.

       * make a minimal shak tape on tape cartridge number 2:
          
	   shak -i 2 newtape 
      
             Now look at the tape index file in $SHAK_INDEXPATH directory.

       * Next, make a full unattended backup:
	  
	   shak run -t fullu backupshak.sh

          type 'fullu' requires a shak tape (i.e it expects
	  to find one or more shak headers on the tape or a
	  pre-existing tape index file for that tape cartridge 
	  number).

      * Next, make another full unattended backup that involves no checks
         and only does a 'mt rewind' and 'mt eom' as a check on
	 the tape labeled 2 :

	  shak -m -i 2 run -t fullu backupshak.sh

       * Other backup types, for hosts larry, bert and
         kermit can all be put on one tape assuming you have
	 such hosts: 
          
	  shak run -t diff -h larry backupshak.sh
	  shak run -t fullu -h bert backupshak.sh
	  shak run -t incr -h kermit backupshak.sh
       
       * for minimal tape manipulation: If the tape number is '2'
	  shak -i 2 -m run -t full backupshak.sh
             
	     this is the same as  
	  shak -mi2 run -t full backupshak.sh

          This command will overwrite the tape and make a new
	  tape index file.

       * to test the tape index file for the currently loaded tape:
	  shak testtape -x 
	     OR
	  shak testtape -b 
	     OR
	  shak testtape
       
       * to look at the values passed to the callers of shak_testtape
	 look at the file produced with the '-o' option.
	 
	 shak testtape -b -o /usr/tmp/any_filename
         cat /usr/tmp/any_filename

            The same result is obtained for the tapeless mode
	    though the tape will not be moved to 'eom', which
	    is the normal and important function of the
	    'testtape' subcommand.
       
	 shak -Q -i 2 testtape -b -o /usr/tmp/any_filename
         cat /usr/tmp/any_filename
       
         Note: a tape number must be given with the 'i' option otherwise 
	 shak will read the tape to determine the tape number.

       * to query the filter date that will be set for use by the backup
         'backupshak.sh' on host 'yourhostname' for the type 'incrs':

	  shak -myy run -t incrs -h yourhostname backupshak.sh

            This assumes that backupshak.sh is in SHAK_USERPATH on host 
	    yourhostname and it is normal to exit with an error.

       * to query the tape number 2 without requiring a tape drive or the
         tape.

	  shak -yymi2 testtape

       * to run a user backup that accepts an argument 'list' which
         lists files and does not write to the tape and uses a mode of 'shak'
	 which does not require a loaded tape and does not write 
	 the tape index file:

	    shak -mQ run -t incrs -h bert backupshak.sh list

              OR

             shak -mQ -D /dev/null run -t incrs -h bert backupshak.sh list

       * to list the tape index file of tape 2 
         without loading a tape:

          shak -Qi2 testtape
         
         OR
	 
	  shak query -p -r --tape 0002

       * run the trivial example backup, trivial.sh. Tell it
         to sleep for 10 seconds but only give it 2 seconds to run.
	  
	  delay_userbackup=2  ./sbin/shak -q run trivial.sh 10

         A message is sent through logger(1) when the watchdog timer
	 expires and some mail is sent to root, this will take your tape
	 unit offline.
	 Try looking in /var/log/messages  and read your mail for
	 root user.

       * run trivial.sh in the background and redirect the stderr file
         of trivial.sh to stderr.out in the local file system.

          shak -qa run -r stderr.out trivial.sh

            Now look at the contents of stderr.out

       * run trivial.sh in the background and redirect the stderr file
         of trivial.sh to stderr.out in the local file system but dont't
	 let it finish.

          delay_userbackup=2 shak -qa run -r stderr.out trivial.sh 10

            Now look at the contents of stderr.out


-- MORE EXAMPLES HIGHLY SPECIFIC TO A PARTICULAR BACKUP
   PROGRAM. In this case user/script/backupshak.sh.
   
   If your backup program is different, then these examples
   don't apply.  Shak doesn't give a hoot what program
   it runs, this allows shak to be a general purpose tool. 
   The examples below all work without a loaded tape.

   Use of the examples below requires the program 'filesize'.
   It can be found in the unsupported/src directory.  It is
   not made by default. To make and install it do:

       	cd unsupported; make filesize
  
   This creates 'filesize' in unsupported/src directory.  To
   use the examples below filesize must be in $SHAK_USERPATH/bin
   directory. The installx make target installs it there so you
   don't have to. Other make options don't even build it.

       * Run the 'test' option of the backupshak.sh
         program.  This will list the exported environment
	 variables. 
       
          shak -Q run -t fullu backupshak.sh test

            This command will display the environment 
	    variables including the filter date SHAK_SHAKDATE.
       
       * Run the 'list' option of the backupshak.sh
         program. This will list the files that would be backed
	 up. 

          shak -Q run -t fullu backupshak.sh list
             OR
          shak -Q run -t incrs backupshak.sh list
       
       * Run the 'sum' option of the backupshak.sh
         program. This will add up the size of the files to
	 be backed up.
          
	  shak -Q run -t fullu backupshak.sh sum

USING BATCH MODE
-------------------------------------------------------------------------

	* Example:  Make a file in $SHAK_USERPATH that contains:

	-t diff -h larry backupshak.sh
	-t fullu -h bert backupshak.sh
	-t incr -h kermit backupshak.sh
	 
	 Call it dailybatch1, then they can all be run with the
	 following command using tape 22 in the example command.
	 (see ./user/script/dailybatch1)

	 shak -i22 -m run -b dailybatch1

	The tape index file is updated just as if they were run separately.
	The advantage is there is no "testtape" operation between each
	backup. (The minimal testttape operation is a rewind and eod (or eom)
	operation) .

        If you want to overwrite the tape add the -x option :
	 
	  shak -i22 -m -x -c250000 run -b dailybatch1

	* Another example:
          Here the first backup overwrite the tape,
	  the remaining backups append.

            -x -t full -h low01 full0.root.all -z
            -t fullu -h low00 full0.src -z
	    -t fullu -h low01 full.usrlocal -z

          Or if you specify -x in the shak command and run a batch file,
	  the overwrite only applies in the first backup of the batch.


USING LOCKING
-----------------------------------------------------------------------
	To enable locking set the 'devicesecure'
	configuration item to "yes".

	Locking is only cooperative, meaning a non-cooperating process
	is able to interfere (get a open) on the tape device.

	To debug locking run something like this:
  shak_debug_locks=yes shak_debugQ=2 /usr/sbin/shak -q run -M trivial.sh 20

	To produce a timeout failure (and a halt in the queue) :
         shak_debug_locks=yes shak_debugQ=2 delay_userbackup=6 /usr/sbin/shak -q run -M trivial.sh 20

	   Watch /var/log/messages for output.


USING STDERR MONITORING
------------------------------------------------------------------------
        STDERR MONITORING function reads then stderr stream of the user backup
	and takes action based on its content.  
	It forms a one-way expect protocol with the user backup.
	The expected content is specified by 
	environment variables containing a regular expressions.  An action
	is performed when one is matched.  The five variables are intended
	to signify a BEGINing, ENDing, a WARNING, receipt of SIGTERM, 
	and a FATAL_ERROR.

	The current implementation is with the expect(1) program 
        (which is a TCL based toolkit for automating interactive programs).

		Variable         Default Value
	    shak_errmon_fatal   I/O|Error|ERROR|error|fail|fatal
	    shak_errmon_begin   __BEGIN__\r\n
	    shak_errmon_end     __END__\r\n
	    shak_errmon_sigterm __SIGTERM__\r\n
	    shak_errmon_warn    "^..*$"
            shak_errmon_timeout1    -1    (means disable)
            shak_errmon_maxwarn    25

        shak_errmon_timeout1 is a number in seconds which is the allowable
	time between "shak_errmon_begin" and "shak_errmon_end" matches.

	The user program simply writes to stderr what is expected.

	This feature is turned on by the -m switch of the `run' subcommand or the
	configuration file.

	Examples:

	Simulate a fatal error, then check your mail for the messages.
            ./bin/shak -q run -m trivial.sh 3 'error'
	
	Simulate a (remote shell) timeout error, then check your mail 
	for the messages.
            delay_userbackup=2 ./bin/shak -q run -m trivial.sh 7

	Simulate a backup timeout error.
	    shak_errmon_timeout1=4 ./bin/shak -q run -m trivial.sh 8

	Make a user backup program and test STDERR MONITORING.
            make a file in /usr/tmp and call it testmon
	    
	    Type the following at the shell prompt:

	    $ cat >/usr/tmp/testmon
#!/bin/sh
trap 'echo __SIGTERM__ 1>&2; exit 1' 1 2 15
echo __BEGIN__ 1>&2
i=0
while [ $i -lt 8 ]; do
sleep 3
i="`expr $i + 1`"
echo "HELLO $i" 1>&2
done
echo __END__ 1>&2
		<Now Hit ctrl-D, this will write the file to /usr/tmp/testmon>

	    Make it executable
            $ chmod u+x /usr/tmp/testmon
            Now run shak 
	    $ shak -Q -i9999 -m run -m -t0 -h localhost /usr/tmp/testmon
              
	      or test maxwarn feature
	    
	    $ shak_errmon_maxwarn=3 shak -Q -i9999 -m run -m -t0 -h localhost /usr/tmp/testmon


MAKING SHAK RUN SETGID/SETUID 
------------------------------------------------------------------------

Preliminary support is included for running shak with setgid/setuid
support.  It is based on proper/src/sush.c

** Actual setup and design for using sush.c and making shak secure for setuid
   operation is not part of this release. **

sush is a program that execs a shell script with the setuid/setgid
permissions of the program file just like a binary program.  sush 
is copied to the name NAME (or linked to 'NAME' ) of the file that is run.  
sush forms  the analog name NAME.sh which is the shell script and executes
it with the right permissions.  NAME (i.e. sush) must be setuid(root)
(chmod u+s NAME). NAME.sh must exist and indicate that it wants to be 
run with setuid/setgid permissions by setting them appropriately 
(even though they have no affect, usually, on shell scripts) since
these settings tell sush what to do.


USING THE QUERY FEATURE
-------------------------------------------------------------------------
EXAMPLES:
     * List all the backup headers. (formated and abridged)
           shak query -r
     
     * List all the backup headers. (unformated, as in index file)
           shak query -r -p

     * List all headers on host "kermit" that have been 
       performed since 1996/01/01 12:03:44 EST

	   shak query -r --backhost kermit --timel "1996/01/01 12:03:44 EST"

     *  List all headers that may be your program 'backupshak.sh'
        performed on host 'ernie'
	 
	   shak query -r --rcsfile backupshak --backhost ernie


     Warning: regular expressiong will match sub-strings, hence
              --type diff will match tyoes 'diff' and 'diffs'.
	      To match only 'diff' use --type 'diff$'.

              Note: --type 'diff\$' will match nothing because
	            the '$' is applied literally and not as a
		    regular expression special character for 
		    "nothing at end of line" which is what is
		    intended. 


RESTORATION OF A BACKUP ON A SHAK TAPE.
-----------------------------------------------------------------------

Shak does not currently support a restoration 
process however since the tape is collection of standard tape
files with ascii text headers it is possible  to use pre-existing
standard commands to locate a backup on the tape.  The shak
command or library is not required.

Here is a example scenario and command sequence:
   You are searching for a backup on the 7th tape file, you'll
   know that when you see (echo to the terminal) the header on 
   the 6th tape file.  Your disk crashed so you lost everything
   including the 'tape index file' stored on the crashed disk.
   (If you are using the no header mode, you are out of luck
   unless you include the tape index files in every backup and
   restore the tape index file.)   

				// I'm assuming the tape device is /dev/nst0
   mt -f /dev/nst0 retension	// optional.
   mt -f /dev/nst0 rewind	// may be optional (if you know it is rewound).
   cat </dev/nst0		// read the header.
   cat </dev/nst0		// read the header, not it.
   mt -f /dev/nst0 fsf		// space over a backup.
   cat </dev/nst0		// read the header, not it.
   cat </dev/nst0 >/dev/null	// same affect as 'mt fsf' above.
   cat </dev/nst0		// read the header, Got it. (6th tape file)
				// The next tape read will be your backup.
   restore_program </dev/nst0   // Your restore program that reads the tape.

--------------------------------------------------------------------
BUG REPORTS, IMPROVEMENTS:

Ideas, comments and improvements and BUG reports 
are welcome at
         jhl@richmond.infi.net

END OF README.
