This file gives the history of Paranoid Backup, explains why Paranoid Backup
is better than other backup programs, and describes some alternative
complicated ways to install it. If that does not interest you, skip this
file and read the file 'quick_start', which explains how to do a simple
installation.

You do know the difference between supplemental/differential/incremental
backups and full backups, don't you? A full backup is where you backup all
files. A supplemental/differential/incremental backup is where you backup
files which have changed since the last backup, sometimes since the last
full backup and sometimes since the last supplemental/differential/
incremental backup. Supplemental/differential/incremental backups are
better, because old versions of files are saved forever, in case you ever
want them again; and because there is less data to back up, which saves wear
and tear on your backup drives, disks, and tapes. However,
supplemental/differential/incremental backups are much more
complicated than full backups, especially when comparing or restoring; so
many people do full backups instead.

Once upon a time I used to backup my computer to floppy disks with Norton
Backup for Windows from Norton Desktop for Windows 2.2. It was difficult to
compare or restore supplemental/differential/incremental backups, so I did
full backups only. I had only one disk set, so I was always overwriting the
previous backup. One day I noticed that a file which I rarely used was
corrupted. I tried to restore this file from backups, but the backed up
version was corrupted too; the backup program had automatically thrown
away the good copy of the file and replaced the good copy with a corrupted
copy. What good is a backup program which automatically deletes my data so
that I can not restore my data?

Later I got a tape drive, and used Seagate/Arcata Backup for DOS, which
came with the tape drive. One time an evil program trashed my hard
drive. I did a restore. The backup program did restore my files, but it
also resurrected hundreds of files which I had once backed up, but later
deleted. It took me two weeks to hunt down and delete all those undeleted
files.

These experiences taught me that most backup programs work so badly, that
most people are wasting their time when they backup their computers.

When I switched to linux I resolved to find a backup program which actually
worked. I started trying some backup programs, but I soon discovered that it
was a lot of work to figure out whether or not they really worked, and how
well they could cope with various possible disasters; and it was neccessary
to create kludges to make them do what I wanted. So I decided that if I
was going to do that much work, then I might as well create my own backup
program.

A lot of backup programs seem to require expensive, high performance tape
drives and expensive, error free tapes. I do not understand this. It seems
to me that if money is no object, you should set up redundant servers in
seperate bomb shelters, like a mirror or RAID configuration, and not backup
at all; if one server fails, use one of the others. On the other hand, if
you do not want to spend a lot of money, you should use a cheap tape drive
with cheap tapes. Paranoid Backup is designed to work with cheap tape drives
with bad performance, with cheap tapes which may contain errors, and to use
as few tapes as possible.

If you do not have a tape drive, it also works with disk drives.

However, I have found that tapes and tape drives are not reliable.
Paranoid Backup works best if you make many small backups; tapes and
tape drives work best if you make a few large backups. Hard drives
are now inexpensive. So I have modified Paranoid Backup to backup to a
directory, which can be on any mounted drive. The tape stuff
is still there if you want it.

Paranoid Backup can detect corrupted backup archives, and can extract as
much usable data as possible.

Paranoid Backup has documentation and error messages which are easy to
understand.

If you replace a file with an older version, Paranoid Backup will consider the
file to have been changed, and will back it up. Most Unix backup programs
select files based on the file date, and will not do this. This may be
helpful if you spoof the date on your computer because you run a program
which is not Y2K compliant, or because you run a demo program which will
only run on certain dates; or if your boss changes his mind about all those
changes he told you to make in your report. Also, if you copy an old file
to your computer from somewhere else, then Paranoid Backup will treat it as
a new file, even though it has an old date. And if you rename a file or
move it to a different directory without changing the date, Paranoid Backup
will consider the file to have been changed, and will back it up (actually,
Paranoid Backup will think you deleted the file and created a new file;
the important thing is that Paranoid Backup will back up the changes you
made). If you change the permissions or owner of a file, it will be treated
as a changed file, and Paranoid Backup will back it up.

Every time you do a backup, Paranoid Backup checks if you have deleted any
files which were previously backed up. If so, Paranoid Backup changes the
default selections so that the deleted files will be automatically
unselected in future compares or restores. The files are not removed from
the old backups and can still be selected for compare or restore. The
purpose of this is so that if you delete a file, you will not be warned that
it is missing in every compare from now until eternity, only from now until
the next backup. Also, if your hard drive crashes and you do a restore, you
will not have to delete the file again.

You can tell Paranoid Backup you are installing new programs, and Paranoid
Backup will keep track of what new files are installed, and will automatically
exclude those files from being backed up. Whether or not this is useful
depends on how to plan to restore your hard drive after a crash. You could
backup data and programs together and restore them together; this is easiest
to set up. But you could also reinstall programs instead of restoring them;
in this case programs should be excluded from backups. While this is more
complicated, it has the following advantages: It uses less tape. Your
programs may be on CDROMs, which last longer than tapes. You may want to
reinstall a newer version of the program. And you may have to reinstall your
programs before doing a restore anyway.

Paranoid Backup can make multiple copies of data in a backup. Obviously
this is redundant, but if your data is important and you are worried that
the tape might degrade, and you might be unable to read the data back, then
you might want to do this.

You can have multiple tape sets, with the same or different selection
criteria, and Paranoid Backup will maintain seperate data for each tape set,
and Paranoid Backup will not think a file is backed up because it was backed
up in the other tape set (which would be a problem in DOS when selecting files
based on the archive attribute).

Paranoid Backup requires an external archive program. In theory it can use
any archive program, but in reality tar, cpio, and afio do poorly at
handling corrupted archives, have incomprehensible error messages, and/or
have poor documentation. Therefore I suggest you use kxarc; if I was
satisfied with tar, cpio, or afio I would not have created kxarc. kxarc
compresses each file individually, like afio; so that if one file is
corrupted it does not prevent kxarc from unarchiving other files, unlike tar
and cpio. Unlike afio, kxarc compresses the file header, which contains the
file name, date, permissions, owner, etc, with the file contents; this makes
kxarc archives a little smaller than afio archives, although it also makes
kxarc slower than afio. When kxarc adds a checksum to a file, the checksum
protects the file header as well as the file contents. When kxarc compares a
file, it compares all the information in the file header first, and it only
compares the file contents if the file header matched. kxarc can skip or
include corrupted files. And kxarc error messages are easier to understand
than tar or cpio error messages, and much much much easier to understand
than afio error messages.

Paranoid Backup is probably slower than other backup programs; speed was not
important to me so I did not worry about it. It may be the slowest backup
program ever written.

Some people do full backups every backup, overwriting the previous backup.
The problem with this is that if a file becomes corrupted but you do not
notice it until several backups later, or if you delete a file and then
decide you want it again several backups later, then the file cannot be
restored.

You could do all your backups to new tapes and save all your old backups
instead of overwriting them, but all those tapes cost you money, and if you
do want to restore an old file, you do not know which backup it is in, and
there is no easy way to search all backups.

You could have two tape sets, and overwrite the older tape set each backup.
If you do compares, the compares should detect corrupted files. However, it
is theoretically possible for a file to become corrupted after the compare
but before the next backup, even if the next backup occurs immediately after
the compare. So to be absolutely sure you are not backing up any corrupted
files, you must compare the last backup, make a new backup, compare the
previous backup AGAIN, and compare the new backup you just did. If these
compares show that the new backup you just did included any corrupted files,
then the new backup you just did is no good, in the future it will be used
to verify the next backup, and for that it must be absolutely perfect; you
must fix the corrupted file and then do everything again. This is
complicated and you still have no way of restoring files you deleted several
backups ago.

On the other hand, some people do one full backup, and then do
supplemental/incremental/differential backups. This makes it complicated to
do a restore or compare or to find and restore a file from an old backup,
but a good backup program will automate the process and make it seem easy.
There are two problems with this. First, files are usually selected for the
supplememental/incremental/differential backups based on the file dates. If
you change the permissions, owner, name, or parent directory of a file
without changing the date, the file will not be backed up even though it has
changed; also if you download a file with an old date it will not be backed
up even though it is a new file. The second problem occurs when you back up
a file and then delete it. If you do a compare, it will warn you that the
deleted files are missing; if your hard drive crashes and you do a restore,
the deleted files will be resurrected. The more backups you do, the more
deleted files accumulate, and the worse the problem becomes.

You could do a full backup, and then do a few supplemental/incremental/
differential backups, and then start again with another full backup. This
does not fix the problems of supplemental/incremental/differential backups;
it only prevents the problems from accumulating; and you still have the
problems of repeating full backups.

All these problems can be fixed by using Paranoid Backup.

When you use Paranoid Backup, you do not choose between full backups and
supplememental/incremental/differential backups. Paranoid Backup does not do
full backups. There are no options for choosing the type of backup. Paranoid
Backup only does one type of backup, a supplememental/incremental/
differential backup where it backs up files which do not match the
descriptions in the virtual restore list. If you use a zero length file as
the virtual restore list, no files will match and all files will be backed
up; this is the equivalent of a full backup, but if all you want to do is
full backups then you do not need Paranoid Backup.

Most people will want to set up one configuration to back up all their
files, but Paranoid Backup allows you to set up multiple configurations. For
example, you could do one backup for /home, one backup for /usr, and one
backup for everything else. Obviously it is more complicated to have several
configurations instead of one, but there are also some advantages. The
different configurations can have different schedules. For example, you
could backup /home every week, /usr every six months, and everything else
every month. If your backup requires two or three tapes, you could split it
up into several smaller backups, and each might only need one tape, and then
you would not have to change tapes in the middle of a backup. Maybe you have
a network, and you want a seperate configuration for each computer.

If your hard drive crashes and you need to do a restore, you have to
reinstall the operating system and reinstall and configure Paranoid Backup
before Paranoid Backup can restore anything. There is no point in trying to
use Paranoid Backup to restore the operating system because you have already
reinstalled the operating system. Since you are not going use Paranoid
Backup to restore the operating system, there is no need to back up the
operating system. That is why the operating system is usually excluded from
backups. However, if you do not want to reinstall the operating system, if
you want Paranoid Backup to restore the operating system with the data, then
you should format your hard drive into two linux partitions, a normal
partition and an emergency partition. Install a linux operating system onto
BOTH partitions. The normal partition is your real linux installation. The
emergency partition can be small; you could use an old linux 1.0 or 1.2
distribution and it probably needs only 20 or 30 megabytes of disk space.
The emergency partition does not need many programs, but it does need perl
and the libraries, and you ought to have emergency utilities and a text
editor as well. Install Paranoid Backup onto the emergency partition. Mount
the normal partition, and use the mount directory as the directory to
backup. Usually you boot to the normal partition to run linux, but when you
want to backup or compare, boot to the emergency partition. If your hard
drive crashes, reinstall the emergency partition, reinstall Paranoid Backup,
reconfigure Paranoid Backup, format and mount the normal partition, and then
use Paranoid Backup to restore everything on the normal partition (and then
run lilo to make the normal partition bootable). Also note that having an
emergency partition can be very helpful if the normal filesystem has
problems; it is much easier to boot from the emergency partition than to
boot from a floppy.

If you want to backup your data files, then it makes sense to exclude your
program files because you will have to reinstall your programs
before you can use paranoid backup to restore your data. This is the
normal way to use paranoid backup. However, you can use paranoid backup to
watch for changes to your files. If you are worried about corruption
or virus infections in your operating system files, then you can include
your operating system and program files in a backup, and
then at any time in the future you can use paranoid backup to compare your
operating system and program files with the old versions, and paranoid
backup will tell you if any of your files are changed, and paranoid backup
can fix file corruption and viruses by restoring the old, correct versions
of the files.

Paranoid Backup does not have to be installed in /root/Paranoidbackup.
Currently it is organized like a DOS program, with its own directory and
with executables, documentation, configuration, and data all in the same
directory. I organized it this way because it was easier to have everything
together while I was working on it. You could make it more unix-like by
moving the documentation to one of the doc directories, and the executables
to one of the bin or lib directories, etc. Probably paranoidbackup.pl should
go in sbin or lib, since only the system administrator should be doing
backups; probably kxarc should go in bin, so anyone can use it; and the
other executables could go in either bin, sbin, or lib. You might think that
Paranoid Backup should not go in /usr, so that you can run Paranoid Backup
without mounting /usr; but Paranoid Backup requires /usr/bin/perl, so you
might as well put Paranoid Backup in /usr. /sbin/backup, /sbin/compare, and
/sbin/restore could be made into links to the backup program. How about
making that part of the linux file system standard? Paranoid Backup makes a
lot of lists of files; since backups are supposed to be done by the system
administrator, these lists should probably be kept in /root, although they
could go in etc, var, or lib. I think it is more organized to have a
seperate directory for the lists for each configuration, but that is not
required.


If you are trying to use linux to backup a dos, umsdos, or
vfat filesystem, then you may find that every time you
switch between standard time and daylight savings time, the last change
time of the files on your dos filesystem will change
by one hour, and paranoid backup will try to backup every file because the
time changed. This occurs because the dos filesystem driver needs
to convert local time to unix standard time. To correctly convert local
time to unix standard time, we need to know if daylight savings time was
in effect at the time being converted, for all
possible times and all possible time zones. However, the dos
filesystem driver does not do this. Instead, the dos filesystem driver
ignores daylight savings time. This means that it is much easier
to convert local time to unix standard time, so the kernel can be smaller,
but times may be inaccurate by one hour. (This only affects
time conversions inside the kernel. User
programs can use library functions which convert time correctly.)

The dos filesystem driver uses tz.tz_minuteswest to convert local time
to unix standard time. tz.tz_minuteswest is set by hwclock, using data
from environment parameter TZ or else from file /etc/localtime. hwclock is
usually run from the boot scripts. If hwclock uses data
from /etc/localtime, and if /etc/localtime includes data about
daylight savings time, then hwclock adjusts tz.tz_minuteswest for whether
or not daylight savings time is currently in effect (man hwclock
says tz.tz_minuteswest is not adjusted for
daylight savings time; man hwclock is wrong) (I checked this with
redhat 7.2 from year 2001; hwclock version
2.4c/util-linux-2.11f, linux kernel version 2.4.19). tz.tz_minuteswest
does not change when daylight savings time
changes; tz.tz_minuteswest changes the next time you boot your
computer or run hwclock after daylight savings time changes.

So if you do not want your file times on your dos filesystem to change
when daylight savings time changes, you need to make sure that
tz.tz_minuteswest does not change. The simple way to do this is to edit
your boot scripts, and set environment parameter TZ before running
hwclock, and then unset TZ after running hwclock. 'export TZ=est5'
will set tz.tz_minuteswest to 300, 5 hours different than unix
standard time, which is correct for USA eastern
standard time. See man tzset for
more information about environment parameter TZ. User programs like ls
will use TZ if TZ exists; that is why you should unset
TZ after setting tz.tz_minuteswest. Also, you should set tz.tz_minuteswest
before you access your dos filesystems; to
be safe, set tz.tz_minuteswest before you mount your dos filesystems.

Or you could reset tz.tz_minuteswest before running paranoid backup.
This might be a good idea if you think that other drivers or parts
of the kernel expect tz.tz_minuteswest to be adjusted for daylight savings
time. In the paranoid backup configuration, put something like:

PreLsCommands=umount /dos;export TZ=est5;hwclock --hctosys;mount /dos
PostLsCommands=umount /dos;hwclock --hctosys --localtime;mount /dos

Unsetting TZ is not needed because paranoid backup will set TZ in a shell,
and TZ will be lost when the shell exits. Remounting the dos filesystem is
needed to clear the disk cache.



The format of the ls lists is different in version 1.2 and 2.0 than in
version 1.1 or 1.0 because version 1.2 uses different ls options, as
explained in the file changes. As a result, if you try to use version 1.2 
or 2.0 to continue a backup sequence which you began with version 1.0 or
1.1, paranoid backup will think that every file has changed, and will
backup every file.

If you really want to continue the backup sequence, you might want to undo
that change by editing the file paranoidbackup.pl and deleting
'--numeric-uid-gid' from the ls command.

If you want to continue a backup sequence which you started
with version 1.x with version 2.0, you need
to convert the old directory exclude file to the new directory include
exclude file.




If you want to save disk space, compress the text files with gzip ('gzip -9
*.text'); and combine paranoidbackup.pl, kxarc, compare_file_lists.pl,
mdti.mdisks.pl, and mdti.tapes.pl into one *.tgz file ('tar -cf -
paranoidbackup.pl kxarc compare_file_lists.pl mdti.mdisks.pl | gzip -9 >
pb.tgz'); and adjust pbui to uncompress it before running, and recompress it
when done.

In theory the mdti servers can be used for other things besides Paranoid
Backup. Therefore, you might want to make a directory named '/etc/mdti',
and make a subdirectory for each disk or tape set. However, probably
the mdti servers will only be used for Paranoid Backup, so it is probably
better to make a directory named 'mdti' in each configuration directory for
the disk or tape set for that configuration.

All of the parts of Paranoid Backup are perl scripts. Originally, Paranoid
Backup was supposed to be a backup script with a file selection program; I
started writing in perl because perl seemed easier to learn than C, and
because perl seemed more appropriate for the string functions in the file
selection program. As Paranoid Backup grew, it seemed easier to continue
with perl than to switch to C. Paranoid Backup would run faster if these
perl scripts were compiled; there are ways to precompile a perl script, and
there are ways to convert a perl script to C. kxarc should be the first to
be compiled, because it takes the longest to compile when it loads, and it
is loaded and reloaded more frequently than any of the other parts. Also
note that it is useful for a backup program to be able to run a restore from
a floppy disk, because then you do not have to reinstall the operating
system before doing a restore. But Paranoid Backup requires perl, and perl
is too big to fit on a floppy disk, so you can not run Paranoid Backup from
a floppy disk; although it might be possible from a zip disk or ls-120 disk.
But if Paranoid Backup was converted to C and compiled as a static binary,
then it might be possible to run it from a floppy disk.

Paranoid Backup has many options for coping with various disasters. This
does not mean the Paranoid Backup is prone to disasters; this is merely an
indication of the paranoia of the author. While I was creating it, I kept
thinking of all these things that could go wrong, so I tried to invent
solutions. You know Murphy's Law: if anything can go wrong, it will. I could
have named it "Murphy's Backup". Most of the disasters that Paranoid Backup
can cope with are very unlikely to occur, so some programmers would not have
bothered to prepare for them.

If someone finds a bug, I may fix it, especially if the bug seems likely to
make trouble for me; but I am not really interested in being the maintainer
of Paranoid Backup. So if you want to take over and be the maintainer, go
right ahead. I think I do a good job of deciding what a program should do,
but I am not so good at actual coding. I created Paranoid Backup to use with
my new tape drive, but it has taken me so long that my "new" tape drive is
now old and obsolete! And I am tired of working on Paranoid Backup.

Kenneth Howlett    av556@detroit.freenet.org
