
GUIDE TO TBACKUP INTERNALS
--------------------------

 Koen Holtman

[Note: this note describes v0.8 and has not been updated since]

This guide documents the internal structure of the tbackup programs.

It is intended for people who want to change or extend tbackup, e.g.
by adding methods.

This guide is not meant to be complete. Information that is easily
obtained by studying the actual tbackup code is not included.  Here I
mainly document the general internal structure and some conventions that
need to be followed to be compatible with future versions of tbackup.
Some pitfalls are also documented.

Don't complain that this file is less easy to read than the manual!
It is supposed to make reading the source easier.

*** PHILOSOPHICAL RAMBLINGS

Tbackup is meant to be an open system, meaning that it should be easy
to add functionality.  Tbackup has some `internal interface standards'
to allow this.  It should be easy, for example, to add a write method
that copies the archive over a network or a choose method that only
backs up files containing the word `kibo'.

However, tbackup is also a non-open system in the sense that only one
person (me) has been maintaining and extending it.  When starting with
tbackup I decided that I would not invite others to develop the code
with me, I feel that a critical piece of software like a backup
program should not be designed by a loosely connected group of people;
this would lead to too many internal inconsistencies.

At this moment (V0.8) the internal interface design is more or less
finished and should accommodate a large range of new methods.  In other
words, adding a method now is more like adding a unix tool than like
extending the unix kernel.

Methods can now be developed and maintained by different persons
without needing extensive communication and synchronization.

On the other size of the coin, tbackup now has so many features
already that the time spent adding a new one may be better spent on
another project.

I have trouble thinking up another feature that could be useful to
many people. 

Maybe one could come up with a choose method that knows about linux
distributions.  For example, with the slackware distribution,
comparing ctimes on the files in /var/adm/packages and last
modification times on the files from the package, one could exclude
all files that came from a package.  The trick is *not* to exclude
package files that were edited by the user after installation.

One could also make incremental backup methods that know about moved
and deleted files.

I feel that tbackup now has the potential to safely grow from a
one-person project into a system maintained and extended by a group of
people.  Of course it remains to be seen if this will really happen.
I've decided to be happy either way.

If you feel like extending or even like maintaining a part of tbackup,
drop me a note.

*** METHODS

The real work in the tbackup, trestore, ... programs is done by the
various methods.

These methods are really `mini programs' that are controlled by
tbackup.  This structure makes tbackup more flexible and extensible
than archivers like tar, which integrate everything into one big
program.

I will mainly describe the method interface of the tbackup program
itself. The interfaces of trestore, tguess, tlist etc are similar.

tbackup provides some services to the methods: it of course executes
them in a cozy environment, but there also is a standard library of
common functions, mainly for user interaction and parameter
processing.  This library is in the /etc/tbackup/bin/paramrouts file.
This file also contains some documentation on the functions.

Tbackup also implements the progress window and part of the index file
functions, mainly through /etc/tbackup/bin/makelogtty.

Tbackup executes three methods.  Here is a nice ascii graph I made
once. It looks like something from kibo's signature.

  tbackup structure
  -----------------

    user supplied parameters -+------------------+
         |                    |                  |
         V                    V                  V
   +---------------+     +-------------+     +--------------+
   | choose method |---->| pack method |---->| write method |---+
   +---------------+  |  +-------------+  |  +--------------+   |
                     file      ^       archive                  |
                     names     |                                V
                               |                              backup
                      files >--+                              medium

The methods communicate with each other through named pipes and by
setting shell variables.  More about this later.

A method is really a set of script files in the /etc/tbackup/bin
directory.  A simple one is the write method `filedev' contained in
the files filedev.wp and filedev.wx (now is the time to look at them).
The most complicated one is the `floppy' write method, contained in
most of the files starting with `floppy' in /usr/tbackup/bin. Do not
look at these files unless you feel like it.  Note that some of the
code in these files is not a good example of shell programming; I
learned shell programming while writing this code.


*** METHOD CATEGORIES

These are several categories of methods, each identified by a letter.

letter  name    used by 
c       choose  tbackup tguess 
p       pack    tbackup
w       write   tbackup
r       read    trestore tlist
u       unpack  trestore
l       list    tlist


*** PARTS OF A METHOD

A method can have up to three parts, each a script executed at
different stages of the backup/restore/... process.

1) p (parameter) part. Optional. Executed after a `method name or
number' question to get parameters from the user. Example: filedev.wp

2) x (execute) part. Required. Executed during the real backup (or
restore, or ...). Does the real work.  The x parts of all methods run
in parallel during the backup. 

The .?x scripts of the read and write methods should not exit until
everything is read or written.  All other .?x scripts should start
background processes that do the real work and exit.

All .ux and .lx (unpack/list method) scripts should remove the file
/tmp/tbup/archive after their archivers have exited.  Removing the
file is the signal for trestore and tlist to clean up and exit.

Examples: filedev.wx, tar.ux.

Complicated x parts like floppy.wx generally call other scripts.

1) f (finalize) part. Optional. Executed after the backup (or
whatever) is done. Never called by tguess. Example: set.cf


*** METHOD PART NAMING

The file

  /usr/lib/tbackup/bin/<method name>.<kind letter><part letter>

contains the <part letter> part of the <kind letter> method named
<method name>.  Example: filedev.wx contains the script for the
execute part of the write method filedev.

The tbackup programs search for the *.?x files when compiling method
lists for the `choose method' questions.  The presence of *.?p and
*.?f files determines if these optional scripts should be executed.

A method is thus `registered' by having a <name>.<kind letter>x file.
The second line of this file should be of the format

#<number># <name>   -- <description>

Where <name> MUST be the method name. Example:

#50# filedev -- write archive to file or device.

<name> and <description> should not contain the # character. The
number of spaces after <name> must be chosen to make the layout of the
`choose method' question look nice. 

The <number> field (normal value is 50) determines the place of the
method in the method list.  The methods are sorted in numeric order to
list the most important ones first.  A lower number places the method
higher on the list.  E.g. `floppy' has the number 10 so that it appears
before `filedev' in the write method list.

When you add a new method, you must run `make lists' in the
/usr/lib/tbackup/bin directory to have it show up in the method lists.


*** TBACKUP METHOD INTERFACE

tbackup follows the following scheme. Read the tbackup script for more
details.

- ask c,p,w method names and execute corresponding *.?p scripts if present.
- create progress tty, descr2k file, print `... in progress'.
- execute c, p, w *.?x scripts. 
  choosemethod and packmethod are supposed to start background tasks.
  writemethod should return if the archive is written.
- execute c, p, w *.?f scripts if present. 
- clean up.

The .cx and .px scripts communicate with each other through the named
pipe /tmp/tbup/choice.  This should be a stream of
file/device/link/directory names.

The .px script writes the archive to the .wx script via the
/tmp/tbup/archive pipe.

The arrangements in trestore, tlist, and tguess are similar.


*** GENERAL METHOD ENVIRONMENT

In general, all scripts in tbackup call each other using the bash
`source' command.  This is more efficient, and sometimes even
required, else variables won't be set in the calling environment.
Only fork a new bash if you really need to.

The methods can use several shell tools, like grep and gawk. See the
file /usr/lib/tbackup/bin/auxlist for a complete list. Using a tool
that is not on the list may cause trouble in a rescue disk situation.

* progress window and backup index.

See the *logtty* scripts in the bin directory.  The .px script should
write progress/index information to the /tmp/tbup/logtty fifo.  The
fifo may be reopened and closed several times (unlike
/tmp/tbup/archive, for example).  See the floppywr script for the
hacks needed to get extra `start of part' information in the index
file.

Normally the index file name is asked after the .wp script has exited.
If the .wp method takes time creating large temp files, like
floppy.wp, it can execute `logttypars' to ask the index file
parameters. This avoids making the user wait.

* working directory

A method script is always executed with /tmp/tbup as the working
directory.  On exiting, the script is expected to set the working
directory back to /tmp/tbup if it changed it.

* temporary files

A method script may make small temporary files in /tmp/tbup. Names
should be sufficiently descriptive as not to clash with those used by
other methods.

* disk space

A read or write method can use the makebulk script if it wants to
create large temp files.  The method should *NOT* rely on sufficient
space being available.  See the floppy.wp script for am example.  Also
see the flowpause program.


*** VARIABLES

Some variables used internally in the tbackup programs, especially if
the methods must do something with them.

--for all programs--

 only_params: nonexistent or set to `y', depending on if the `pars'
              option is given.

--for tbackup--

 rmbulkdir: set to "echo -n" on startup.  Sourceing `makebulk' sets it to
            "rm -r $bulkdir".  This string should be executed in every
            `aborting tbackup' routine.

--for trestore--

rawdevice: values: "y" or "n".
  "y" means that we are reading directly from a raw device that doesn't
  produce an EOF at the end of the archive.  If set to "y", the tar
  unpack method treats a block of zero's as the end of an archive.
  The rawdevice flag must be set by the *.rp scripts.

--------------------------------------------------------------------------
