















            RR--UUnniixx:: AA uusseerr--lleevveell iimmpplleemmeennttaattiioonn ooff rreemmoottee UUNNIIXX sseerrvviicceess

               IImmpplleemmeenntteedd bbyy OOlleegg KKiibbiirreevv ((oolleegg@@ggdd..ccss..CCSSUUFFrreessnnoo..EEDDUU))
                          aass aa ccllaassss pprroojjeecctt ffoorr CC..SSccii 117733


























                    California State University in Fresno, 1995























                                        - 2 -



                                      AAbbssttrraacctt

            R-Unix  is an extension to UNIX operating system that allows
            program's  use  of  system  resources,  such  as  filesystem
            objects  and  IPC facilities, to be intercepted and serviced
            by a  user-level  program,  possibly  running  on  a  remote
            machine.  This  can  be  used to develop network-distributed
            applications as well  as  represent  arbitrary  objects  and
            databases  as  conventional  files  or  IPC  handles. R-Unix
            library uses dynamic linking to allow any existing  binaries
            run  unchanged  and  use  remote  services.  The  project is
            therefore  is  easily  portable  and  requires   no   kernel
            modifications.  This  distribution  can compile under Linux,
            SunOS 4.x or SVR4.

            At the time of this release, only filesystem code of  R-Unix
            is implemented.


                                     BBaacckkggrroouunndd

            There  have  been an explosion of enhancements added to UNIX
            by various  vendors  and  research  groups.  Most  of  those
            additions  -  like  TCP/IP,  NFS  or termios - were added by
            creating  more  subdirectories  in   kernel   source   tree.
            Undoubtfully  these  features made UNIX more useful (and fun
            to use). But they also resulted in slower and  memory-hungry
            kernels. While original version of UNIX could run in 64K RAM
            of PDP-11 and BSD 3.0 was usable on a VAX  with  2M  and  10
            users logged in, even a single-user workstation running SVR4
            crawls unless it's given at least 32M.

            One  part  of  the  problem   is   implementing   additional
            functionality as part of the kernel. User level programs and
            libraries can be developed  incrementally,  based  on  other
            programs,  libraries  and  kernel  facilities. Two important
            kernel services that user-level code  can  use  are  virtual
            memory  and  dynamic  resource allocation. The first ensures
            that only portions of the program that are  frequently  used
            are  retained in memory, while, for example, seldomly called
            initialization and error handling code is swapped out  until
            it's  actually called. The second lets a program use only as
            much memory, file handles and IPC buffers as it needs during
            a particular run.

            By  contrast, all the kernel code is permanently resident in
            memory and on many systems is required to allocate resources
            statically  at  startup  time. It doesn't have access to any
            libraries and is often  restricted  in  which  other  kernel
            procedures  it  can  call. It's common for kernel hackers to
            reinvent the wheel and  if,  for  example,  kernel  supports
            filesystem  compression  and compressed SLIP, it's likely to
            have two different versions of compression code.










                                        - 3 -


            RR--UUnniixx is my attempt to augment  system  services  by  user-
            level  code.  For now it supports remote file access similar
            to RFS. If I have some free time to work on it,  I  plan  to
            add transparent remote execution and distributed SysV IPC.

            Current  code  is certainly less efficient than NFS. However
            if I ever had to design a production version, I know  a  lot
            optimization I could do. Also R-Unix is more flexible in the
            sense that it's easy to substitute my R-Unix server  with  a
            program that provides access to something other than a plain
            filesystem. For example, one could  design  an  NNTP  server
            that supports USENET access with commands like:

            lleessss
            //@@nnnnttpp//ccoommpp..ooss..lliinnuuxx..ddeevveellooppmmeenntt//22334455__AANNNNOOUUNNCCEE__kkeerrnneell__11__33__88__iiss__oouutt

            and

            eemmaaccss //@@nnnnttpp//nneeww__aarrttiiccllee

            But  anyway,  this is a research project and it's under GPL.
            If you like it, send me e-mail. If you want  more  features,
            add  them and share your changes with me and the rest of the
            net. If you think is useless, well at least it's  worth  the
            price.  By  the  way,  I  don't  know  if  I'll have time to
            maintain it on regular basis, so if you want to release your
            own improved version, really, go ahead :)





































                                        - 4 -


                                   RR--UUnniixx aatt WWoorrkk

            Let's  assume  just  for  a  moment  you  already managed to
            compile R-Unix on two machines, hostA and hostB. Here is how
            to actually use it:

            11))  SSeett  uupp  ppaasssswwoorrddss::  R-Unix  uses  simple  magic  cookie
            authnication. After clients connect to the server, they send
            the  first  32  bytes  of  the  file named .runixkey_clnt in
            user's home directory. In turn, R-Unix server reads  a  file
            named  .runixkey_svc  in  the  home directory of remote user
            specified by a client. If the first 32 bytes of these  files
            don't match, the connection is dropped.

            On  hostA, "vi $HOME/.runixkey_svc" and type a line or so of
            random characters. Then, create ".runixkey_clnt" in the same
            manner.   Now   ftp   to  hostB  and  put  .runixkey_svc  as
            .runixkey_clnt on hostB and vica versa. You  can  also  link
            these  two  files  together  at  the  cost  of slightly less
            security. In any case, chmod both files to 0600  unless  you
            don't mind everyone at the world browsing your account. Also
            both key files must be at least 32  bytes  long  or  nothing
            will work.


            22))  SSttaarrtt  tthhee  sseerrvveerr::  When  you build the program, R-Unix
            server is compiled in "runixd" subdirectory.  Assuming  that
            you install it to somewhere in your $PATH, just type:

            hhoossttAA%% rruunniixxdd --pp <<IIPP ppoorrtt>>

            After a couple of seconds, do "ps -x" to check if the server
            is indeed running. If it died,  the  error  message  can  be
            found  in  system  log  file,  usually  /usr/adm/messages or
            /var/log/syslog. Or you can try -d to see  why.  Here  is  a
            summary of options:

            --pp  ppoorrtt    - specifies an IP port on host machine that will
            be later used by clients to find the server.  If you are not
            very  familiar  with  TCP/IP  networking,  choose  a  number
            between 1025 and 60000. If the  server  dies  it's  probably
            because there is already another service running on the same
            port. In this case, /usr/adm/messages will have  some  error
            message  similar  to  "Address  in use". For this matter, it
            could be another runix server that you started before. Do ps
            -x  and kill off any runixd processes you don't want. If the
            problem persists, try another port number.

            On some systems, there is a default port  used  without  -p.
            But  if  several  users  start  runixd  on the same machine,
            they'll still have to choose different ports.

            --dd        - Normally runixd puts itself into background  and
            runs until killed (even after you log out). With -d however,
            it will stay in foreground and  exit  after  it  serves  one









                                        - 5 -


            client.  It  will also print error messages to stderr rather
            than system log file. Use for troubleshooting.

            --nn        - Non-privileged mode. Normally, if runixd is  run
            by  root,  it  can serve all users on the system that create
            .runixkey_svc files in their home directories. Therefore  if
            runixd  is  started during system boot from one of rc files,
            users don't need to run their  own  daemons.  If  runixd  is
            started by a user, however, it will serve this user only. -n
            forces single user mode when runixd is started by root or is
            setuid root. In the later case setuid privilege is dropped.

            --ff ccooookkiiee__ffiillee   - Specify a cookie file to use  instead  of
            $HOME/.runixkey_svc. Implies -n.

            33))  RRuunn  cclliieennttss::  To make life more interesting, lets login
            to hostB first. Now you have to specify where is your server
            by    setting    an   environment   variable   RUNIXSVC   to
            host:login:port. For example if your  name  is  John  Smith,
            your  login  name on hostA is jsmith but the one on hostB is
            john and you started runixd -p 8612, type:

            hhoossttBB%% sseetteennvv RRUUNNIIXXSSVVCC hhoossttAA::jjssmmiitthh::88661122

            Or if  your  login  shell  is  /bin/sh  or  a  Bourne  shell
            compatible like bash or zsh:

            hhoossttBB$$ RRUUNNIIXXSSVVCC==hhoossttAA::jjssmmiitthh::88661122;; eexxppoorrtt RRUUNNIIXXSSVVCC

            You can omit some components of the address. Missing host is
            assumed to be "localhost", missing  name  is  replaced  with
            your  login  name  on client machine and missing port stands
            for default R-Unix port listed in /etc/services, if any.  So
            if John Smith wanted to start a clients on hostA instead, he
            would just use:

            hhoossttBB%% sseetteennvv RRUUNNIIXXSSVVCC ::::88661122


            Next, you need to specify that R-Unix  shared  libraries  be
            loaded  when  you  run  programs.  For  example, if you just
            compiled R-Unix in your home directory, you can use:

            hhoossttBB%% sseetteennvv LLDD__PPRREELLOOAADD==$$HHOOMMEE//rruunniixx110000//lliibbrruunniixx..ssoo..11..00..00

            Now any programs you run will use R-Unix services.  To  make
            sure  shell  builtin  commands can use R-Unix as well, start
            another copy of your shell first:

            hhoossttBB%% ttccsshh
            hhoossttBB%%

            If you see any error messages, check again that  the  server
            on  hostA  is  running,  RUNIXSVC  and  LD_PRELOAD  are  set
            correctly and my program doesn't have bugs :) Otherwise  you









                                        - 6 -


            should  be able to access the entire directory tree of hostA
            under  /@. For example:

            lleessss //@@//eettcc//ppaasssswwdd  - edit a file on hostB

            ccdd //@@//eettcc
            vvii ppaasssswwdd            - another way to do it

            llss //@@                     - list root directory on hostA

            ccaatt << //vvmmlliinnuuxx >> //@@//ddeevv//ttttyypp22 - bother a user on hostA,
                                          if  chosen  remote  login  can
            write on ttyp2.

            dduummpp 00ff -- //@@//ddeevv//rrssdd00aa >> HHOOMMEE//hhoossttAA..dduummpp -  why not?

            You  can also create and use remote named pipes to implement
            some communication between
            local and remote processes. With  few  exceptions,  you  can
            access  hostA  files  just  like local. Some things to watch
            out:

            ^oo^ Your access permissions on  remote  host  are  that  of
            remote  user specified in RUNIXSVC, which might be different
            from hostB.

            ^oo^ You can't execute remote programs  yet.  This  will  be
            implemented  in  a future release. You can run shell scripts
            with something like sh /@/etc/script

            ^oo^ Symbolic links and remote path names can't  cross  host
            boundaries. For example, /@/etc/../.. is the same as /@, not
            / and  /etc/../@/etc doesn't work as expected. Enjoy!

            ^oo^ Remote calls are non-interruptable

            ^oo^ setuid and statically linked executables ignore  R-Unix
            services.  Watch  out  for  statically  linked /bin/sh under
            SunOS 4.x.

            44)) CClleeaannuupp:: It's best to first exit all R-Unix  clients.  If
            the  server  goes down first, they'll get errors when trying
            to access remote resources. exit the shell  you  started  on
            hostB  and  unset LD_PRELOAD. After that, login to hostA and
            kill the server.

                                  RR--UUnniixx oovveerr TTeerrmm

            tteerrmm is a  serial  line  multiplexor  that  allows  multiple
            TCP/IP  connections  over  modem  without  need  for  kernel
            modifications or root access  at  either  end  of  the  line
            (unlike  SLIP  or  PPP).  It's easy to use R-Unix over term.
            For example, suppose you started a server on remote  machine
            as:










                                        - 7 -


            rreemmoottee%% rruunniixxdd --pp 77889911

            To run clients on your local machine, do:

            llooccaall%% ttrreeddiirr 77889911 77889911
            llooccaall%% RRUUNNIIXXSSVVCC==::::77889911

            Then,  set LD_PRELOAD and run clients as usual. You may want
            to kill tredir process after you are done using R-Unix. It's
            expected that remote file access over a serial line is slow,
            especially with 2400 baud modem. By the way,  /bin/ls /@/etc
            is  much  faster  than  /bin/ls -F /@/etc, because the later
            will stat() each file.

                                     VVaappoorrwwaarree

                 Here are some extensions that I think will be useful in
            future  versions. Suggestions and especially C code are very
            welcome:

            MMuullttiippllee sseerrvveerrss:: For serious use, R-Unix needs  to  support
            connecting  to  multiple  servers  and multiple users on the
            same server.  For  example,  one  could  provide  access  to
            pathnames in the form:

            //@@//uusseerr@@hhoosstt//mmyy__rreessoouurrccee//ddiirr//ffiillee

            This  would  require  a  map  file  similar  to  one used by
            automount, with port numbers for each service and passwords.
            Also,  in  this  case  R-Unix  has  to connect to servers on
            demand, rather than immediately after the program is started
            and handle server failures gracefully.

            RReemmoottee  eexxeeccuuttiioonn::  The  planned semantics is that if a user
            runs /@/bin/prog, the program is started on _r_e_m_o_t_e _h_o_s_t, and
            use  a  temporary  one-shot  server process on local host to
            access local files. This requires integrating server into R-
            Unix   shared   library  and  support  for  remote  terminal
            handling. This semantics would make it very easy to  develop
            distributed applications so I may work on it first.

            MMoorree  ttrraannssppaarreenntt sseemmaannttiiccss:: It's reasonable to support non-
            blocking I/O, some fcntl calls and select but I did't finish
            it yet :)

            OOtthheerr  IIPPCC  ddoommaaiinnss::  Current implementation doesn't include
            record locking, use of UNIX domain sockets and SysV  message
            queues  and  semaphores. It shouldn't very difficult to make
            these services accessible from remote as well.

            FFiillee ccaacchhiinngg:: Currently R-Unix library  requests  data  from
            remote  system  on  any  I/O  call from the program. This is
            clearly unacceptable for slow connections like serial lines.
            It  shouldn't  be  difficult  to cache data for files opened
            read-only on local filesystem, although  it  may  mean  that









                                        - 8 -


            changes  to  original file on the server are not immediately
            seen by all clients. The same holds for stat() results.

            LLiigghhttwweeiigghhtt sseerrvveerr: At this time,  when  a  client  fork()s,
            server  will  fork()  as  well.  This  is the easiest way to
            implement UNIX semantics, but it's clearly inefficient. It's
            possible to redesign
            the  server  to accept requests from multiple clients in one
            process without changing client library. In this  case,  the
            server must remember uid, gid and current directory for each
            client as well as chop I/O requests  into  small  chunks  so
            that one client can't block it for a long time.

            SSttaatteelleessss  sseerrvveerr::  Stateless  servers  are well known to be
            more robust and create less load on server machine, which in
            turn  allows  it  to have more concurrent clients. However a
            stateless  filesystem  server  is  difficult  to   implement
            without  _s_o_m_e  support  in  the kernel. In particular, there
            needs to be a call to open a  file  given  it's  device  and
            inode number instead of name.

            BBeetttteerr  sseeccuurriittyy::  Current  cookie  file authnication can be
            easily subverted by  anyone  who  has  root  access  to  any
            machine where a user runs R-Unix client or server or is able
            to snoop on network traffic. At  the  same  time,  it  seams
            unacceptable  to  ask user for password every time she wants
            to start an R-Unix client. It's probably best  to  use  some
            third party authnication scheme like Kerberos.

                                  CCoommppiilliinngg RR--UUnniixx

            R-Unix  compiles  out  of the box under SunOS 4.x, Linux and
            Solaris 2.x. Solaris port, however, is not  stable  yet.  In
            particular  it  doesn't  work  with programs that are linked
            with -lnsl. I  wasn't  able  to  debug  it  because  current
            version  of  gdb  doesn't  work  well under Solaris 2.4. For
            Linux, you should install libc  5.2.3  and  ld.so  1.7.4  or
            later.  Earlier  versions have bugs that prevent R-Unix from
            working. Current version has bugs too,  but  I  work  around
            them.  From  that, you can guess that R-Unix requires an ELF
            system. Indeed it doesn't work with a.out
            binaries  :)

            You need:

            ^oo^ gcc V2.7.0 or later, except under SunOS  4.x,   2.2  or
            later  will work.

            ^oo^ GNU Make

            ^oo^  new rpcgen. Under Solaris 2.x that's what you get with
            the system. Under Linux, Slackware distribution does include
            new  rpcgen but it has some bugs and I wasn't able to use it
            (it will sometimes generate incorrect function  prototypes).
            Under SunOS 4.x you'll probably need to compile it yourself.









                                        - 9 -


            tirpc2.3 source code is released for public use by  Sun  and
            is  available from ftp sites around the world. I took rpcgen
            source from there and bugfixed it enough  to  use  it  under
            SunOS  4.x  and Linux. For your convinience, modified source
            code and manual are available in "rpcgen" subdirectory of R-
            Unix distribution.

            ^oo^  "!  " command. Depending on your shell, you may or may
            not be able to do:

                 iiff !! ffoooo;; tthheenn
                      bbaarr
                 ffii

            If you don't have builtin "!", please copy "!" shell  script
            in the root directory of R-Unix distribution to somewhere in
            your $PATH. It may even be a nice addition to your /usr/bin.

            ^oo^ In top level Makefile, there is a line that reads:

            zzsshh  --cc  ''ffoorr ii iinn **..aa;; ddoo [[ mmaakkeettiimmee --nntt $$$$ii ]] |||| $$((RRAANNLLIIBB))
            $$$$ii |||| ttrruuee;;

            The idea is to only do  ranlib  if  a  library  is  changed,
            because  some versions of ranlib are very slow. If you can't
            find any shell that has -nt test  option,  just  change  the
            line to:

            rraannlliibb **..aa

            You should be able to build R-Unix now by typing "make svr4"
            for Solaris 2.x  or  just  "make"  for  other  systems,  but
            depending  on exact version of your system and utilities you
            may still need to change a couple of  lines  somewhere,  who
            knows.

            For best results you should create an empty directory called
            /@ on all hosts where you plan to run clients.  If  you  are
            not  root, don't worry, R-Unix will work without it as well.
            The only time it makes a difference is  when  a  program  is
            reading  root  directory (/), because R-Unix will not modify
            it's contents to include @. Therefore without  existing  /@,
            find  /  -type f -print will never enter remote directories.
            find / /@ -type f -print shoud work.

            In addition, under SunOS 4.x you need create  a  link  named
            _libc_so  that  points  to  the version of C library you are
            using. For example:

            ssuunn## ccdd //lliibb
            ssuunn## llnn --ss lliibbcc..ssoo..11..88 __lliibbcc__ssoo

            If  you  don't  have  write   permission   on   /lib,   edit
            preload/dlconfig.h    and   replace   "/lib/_libc_so"   with
            something like "/home/snowcat/lib/_libc_so" so that you  can









                                       - 10 -


            make  a   soft  link  in  some  directory  where you do have
            permission.

            If you want to  set  up  a  default  port  that  runixd  and
            librunix.so  use  when  none  is specified by the user, edit
            /etc/services and add an entry for runix. For example:

            rruunniixx           88338899//ttccpp
























































                                       - 11 -


                           PPoorrttiinngg RR--UUnniixx ttoo aa NNeeww SSyysstteemm

            R-Unix should be  easy  to  port  to  systems  that  support
            programming  interface  to  dynamic  linking.  Here are some
            places that are likely to need change:

            pprreellooaadd//ddllccoonnffiigg..hh:: This header file  describes  details  of
            dynamic  linking  on  supported  systems.  Basically, R-Unix
            needs to be able to redefine C  library  functions  such  as
            open  and  read and at the same time be still able to access
            original functions that are in effect  when  librunix.so  is
            not  loaded.  The most efficient way to do this is currently
            supported by SVR4 - dlsym(RTLD_NEXT, "func") will return the
            definition  of  function  from the next shared library after
            the caller. This  approach  allows  several  libraries  like
            librunix.so to be loaded at the same time. SunOS 4.x doesn't
            support RTLD_NEXT and one must open C library explicitly  to
            get  the  original symbol definition. To port librunix.so to
            other systems, define OPEN_LIBC to return a handle that  can
            be used with dlsym to get original symbol definitions.

            It  may  be  possible  to  support platforms that don't have
            dlopen/dlsym or  equivalent  functions  by  using  syscall()
            interface.  However  it's  not portable either, because it's
            hard to tell which library functions directly invoke  system
            calls.  For  example,  under  SunOS readdir() actually calls
            getdents(). Good luck!

            LIB_INIT and LIB_INITFUNC macros define a way to  declare  a
            function  that gets called automatically when the library is
            loaded. On some platforms, gcc  2.7.0  proved  an  attribute
            "constructor"  that does just that. If it's not available, I
            suggest that you use the same  way  as  currently  used  for
            sunos4.

            Some  systems  define a second set of function names used by
            the C library itself internally. For  example,  under  Linux
            user programs use normal "read" function, but libc functions
            like fopen call "__read "instead, which is normally the same
            as  "read".  The  goal  is  to  make  sure that if a program
            defines it's own "read" function  it  can  still  use  stdio
            functions  like  fgets().  For the purpose of R-Unix though,
            this second alias function must be  redefined  as  well.  If
            your   system's   libc   works   in  the  same  way,  define
            ADD_LIBC_ALT_PREFIX macro to return the alternative function
            name given normal name as an argument.

            rruunniixxdd//ggiivveeffdd..cc  aanndd  ssvvcc//ggeettffdd..cc::  Passing file descriptors
            between processes is not standard and although  most  modern
            unices  do  implement it, each one does it in different way.
            See the "R-Unix  internals"  section  for  more  information
            about how R-Unix uses fd passing.

            pprreellooaadd//MMaakkeeffiillee::  Your  system  may have a different way of
            linking shared libraries. Check it out.









                                       - 12 -


            ccoommmmoonn//aannssiipprroott..hh:: This header defines ANSI  prototypes  for
            all  functions  intercepted by librunix.so. These prototypes
            must _e_x_a_c_t_l_y match those in your system  header  files  (for
            example  "char  *" is not the same as "char const *") or the
            library will not compile successfully. Don't ask me, gcc  is
            just very picky.


                                 RR--UUnniixx  iinntteerrnnaallss

            Should  you want to modify R-Unix, you may find this section
            useful.

            11)) RR--UUnniixx hhaannddsshhaakkiinngg pprroottooccooll:: Clients  always  connect  to
            the  server  using  a  single  port number specified with -p
            option to runixd. This allows R-Unix to be used over a  term
            connection  without  any changes. After connecting, a client
            sends some information in a fixed  structure  that  contains
            client's  version,  procedure  number, desired user name and
            password.  Procedure  can  be  PROC_INIT  (0)   for   normal
            connection or PROC_FORK(1) - a new process making additional
            connection after fork. In both cases server returns a single
            byte,  which  is 0 if the request is sucessful and otherwize
            contains error code.

            This initial handshaking is done without help  of  RPC.  The
            reason is that for security purposes,
            no  other procedure can be called before successful init and
            I didn't want to have a complety separate rpcgen source file
            just  for  handshaking.  The  data is, however, exchanged in
            portable network byte order.

            After successfull PROC_INIT, both client and  server  create
            an  RPC  handle  for  the  connection  and  later all client
            requests are done through RPC calls.

            Source code: runixd/auth_client.c, runixd/serve_client.c and
            clnt/runix_init.c

            22))   RR--UUnniixx   RRPPCC  pprroottooccooll::  rpc/runix_prot.x  defines  the
            protocol used for making remote UNIX requests. Many  of  the
            functions  are  direct  wrappers around the system call. For
            example, an runix_open results in server  opening  the  file
            and  returning a handle to client process for future read()s
            and write()s. But sometimes the interface  is  less  direct,
            for example runix_opendir fetches the whole directory before
            returning to the caller, which then emulates UNIX  semantics
            by having readdir() scan the array and return one entry at a
            time.

            runix_prot.x is compiled with rpcgen, but server and  client
            stubs  generated  are  not  used  directly. Instead they are
            piped through sed scripts in  tools/  subdirectory  to  make
            them  suitable for existance in R-Unix library. For example,
            runix_prot_svc.c looses it's  main  and  has  some  of  it's









                                       - 13 -


            static functions changed to be global instead.

            Source  code:  rpc/runix_prot.x,  svc/runix_server.c (server
            itself), tools/* (conversion)

            33)) rr__ ffuunnccttiioonnss:: The first level of abstaction   on  top  of
            raw  rpc  calls are r_ functions. These are wrappers that do
            remote a remote procedure call, decode the result into  UNIX
            variables  and  data structures and free up memory temporary
            used by RPC. r_ functions  serve  to  convert  RPC  protocol
            semantics  to  UNIX semantics. For example r_opendir() makes
            runix_opendir RPC call that  returns  the  whole  directory.
            Later r_readdir traverses the data structure and returns one
            entry at a time and finally r_closedir free()'s  the  memory
            without sending any request to the server.

            r_  functions  can  be  also  used  directly for testing RPC
            protocol only. There is a header file "clnt/stealth_runix.h"
            that  defines  open  as  r_open,  read  as r_read and so on.
            Source code of a program  can  be  edited  to  include  this
            header  and  call  r_runixinit(NULL);  and later linked with
            -lrunix_clnt -lrunix. As a result, the program will make RPC
            calls  instead of normal C library calls when run. Actually,
            I didn't use this header file since I got RPC stubs  working
            so  it  may be out of date. But it should be easy to make it
            work again.

            Source    code    (for    different    function    classes):
            clnt/clnt_trivial.c clnt/clnt_dir.c clnt/clnt_mmap.c

            Look in clnt_mmap.c for an example of ultimate cheating.

            44)) LLiibbrraarryy iinniittiiaalliizzaattiioonn:: On operating systems that support
            it, R-Unix library is initialized at the time librunix.so is
            loaded.  On  other  platforms, initialization code is called
            the first time  one  of  library  functions  intercepted  by
            librunix.so  is  used.  In  any  case,  the  first  step  in
            initialization is finding addresses of original  definitions
            for  standard functions, such as read(), that are redefined.
            The addresses  are  placed  in  c_  function  pointers.  For
            example  c_read is a pointer to the "previous" definition of
            read, probably in libc or another library that works in  the
            same way as librunix.so.

            The   next   step  is  allocating  memory  for  global  data
            structures used by R-Unix and making  a  connection  to  the
            server.  At  the  end  of  _librunix_init(),  there  are two
            special file descriptors defined: rpc_fd, which is connected
            to R-Unix server and dummyfd that is described later.
            An attempt is made to prevent user programs from thinking it
            can use those two fd's by setting RLIMIT_NOFILE to  be  just
            below them.

            Source code: preload/libinit.c










                                       - 14 -


            55)) FFuunnccttiioonn ssttuubbss:: All of functions redefined by librunix.so
            need to do some of the same initialization and cleanup  code
            that  have  been  separated  into  a  separate  layer.  Stub
            functions have the same name  as  ones  in  the  standard  C
            library.  Therefore,  when  a  user  program  (or  C library
            itself), calls open(), it will execute code in  librunix.so,
            not  in  libc.so. Each of stub functions first checks if the
            library is disabled.  If  so,  it  will  call  the  original
            function,  c_open(). Otherwize, it masks all the signals and
            _s_e_t_s the disabled flag. i_ function is then called to do the
            real work, disabled flag is cleared, signal mask is restored
            and the result of i_ function is returned to the caller.

            The reason for setting the  disabled  flag  is  that  R-Unix
            library itself must call some of the functions it redefines.
            My own code uses c_ functions whenever  possible,  but,  for
            example, RPC code is already in a system library and I can't
            change what it does. If these calls  were  to  re-enter   i_
            functions,  an  infinite  loop  would  result. Similary, the
            reason for disabling signals is that some of i_functions and
            RPC calls are not re-entrant.

            In  some  cases, UNIX allows a system call to be interrupted
            by a signal. When an i_ function decides that it  will  call
            original  c_  functions,  it  clears  disabled  flag and re-
            enables signals before calling it. Stub functions check  for
            this situation and can handle it correctly.

            Source code: preload/libstubs.c

            66))  ii__  ffuunnccttiioonnss aanndd lliibbrruunniixx ddaattaa ssttrruuccttuurreess:: i_ functions
            handle standard library calls redirected  by  R-Unix.  Their
            design  goal  is  to  preserve  UNIX semantics as closely as
            possible.  One  important  problem  is  management  of  file
            descriptors,  as  the library must know which ones are local
            and which ones are remote and also convince the user program
            that all of them exist.

            A  global  integer array, _f_d_m_a_p, is allocated at startup and
            is as big as the number of concurrent open files per process
            supported  by the system. Initially it's all 0. If it's non-
            zero for file descriptor i, then fdmap[i] is the remote file
            handle for this descriptor that is usable with r_ functions.
            fdmap entries are set when remote files are open and cleared
            when  they  are  closed.  dup and dup2 may also change fdmap
            values.

            For each open remote file, there is a local file  descriptor
            used  as a placeholder. It's created by duplicating _d_u_m_m_y_f_d,
            which is opened at startup as an unbound  socket  and  using
            the duplicate's fd number as a local number for remote file.
            This prevents locally created fd's from having numbers  that
            are  already  assigned  for remote files. It also makes sure
            that any  functions  that  are  not  redirected  by  current
            release  of  R-Unix  fail  when faced with "remote" file, as









                                       - 15 -


            only few system calls can be applied to an unbound socket.

            Directories have to be handled in a different  way,  because
            UNIX standard directory access functions - _o_p_e_n_d_i_r, _r_e_a_d_d_i_r,
            etc  -  return  a  data  structure  that  varies  from   one
            implementation  to  another.  Therefore  this data structure
            can't be constructed by a  portable  library.  Instead,  the
            user  program  is  returned  a  result of c_opendir("/tmp"),
            which convinces most programs that they are  indeed  dealing
            with  a  valid directory. /tmp is chosen to limit the damage
            in an unlikely case  if  a  priviliged  program  executes  a
            system-specific  call  unknown  to  R-Unix that modifies the
            filesystem.

            R-Unix itself  manages  remote  directories  by  maintaining
            another  global  array,  _d_i_r_t_b_l.  It is as large as fdmap to
            allow for largest possible  number  of  concurrently  opened
            directories and contains a table of addresses for all remote
            DIR structures active at the moment and their  corresponding
            remote  handles to be passed to r_readdir. Currently, linear
            search is done on every call to readdir and other  directory
            access  functions.  It's my guess that a program is unlikely
            to have many directories open at the  same  time.  But  it's
            easy to change the program to use binary search if needed.

            The  third  global  variable,  _r_e_m_o_t_e, is set by chdir() and
            fchdir() to non-zero when  a  client  changes  to  a  remote
            directory.  open()  and other functions that take a pathname
            consider the file to be remote if (this variable is set  and
            pathname  is  relative  (like  ../dir/file)) OR (pathname is
            absolutely remote (/@/dir/file)).

            Source     code:     preload/lib_i.c     preload/lib_i_dir.c
            preload/lib_i_exec.c

            77)) DDuupplliiccaattiinngg rreemmoottee rreessoouurrcceess oonn ffoorrkk(()):: Currently, R-Unix
            uses a  simple,  although  inefficient  method  of  handling
            fork().   When  a  client  fork()'s,  niether  client's  nor
            server's R-Unix state changes, but the  server  fork()'s  as
            well   and  child  client  processes  forms  an  independant
            connection with the new server. This  approach  conviniently
            handles  maintaining  two separate sets of file descriptors,
            current directories and umask(). However it seriously limits
            the  number of clients one machine can serve. I just thought
            it would be better to implement a stupid  server  first  and
            make it work before working on optimization.

            The handling of fork() is complicated by the requirment that
            R-Unix works with term (this could be one of the most useful
            features before more advanced functionality is implemented).
            Clients, therefore, must connect to a server using a  single
            IP  port  that  can  be  used with tredir and server may not
            connect back to clients.











                                       - 16 -


            A process that is about to fork()  ((aa)) first connects to  R-
            Unix server using the same port as newly started clients. It
            proceeds with handshaking and specifies PROC_FORK instead of
            PROC_INIT.  In  response,  server  prepares  to  pass a file
            descriptor for this new connection  and  returns  client  an
            opaque  32-bit  handle  that  can  be  used,  in OS-specific
            manner,  by  another  server  process  to  get   this   file
            descriptor ((bb))

            The  next step is an RPC call using existing connection of a
            client that passes this handle to client's  server  process.
            The  server fork()'s and the child gets the new connection's
            file descriptor and signals intermediate server  process  to
            exit  ((cc)).  This  new file descriptor is used to service RPC
            calls.

            Now there is a single client process that has connections to
            both  server processes that are created by remote fork() and
            therefore share file descriptors and have  the  same  inital
            UNIX  state (such as current directory, umask and user/group
            membership) ((dd)). Finally the client itself fork()s and  each
            of  two  resulting  processes uses one connection and closes
            another ((ee)).

            Source code: runixd/givefd.c svc/getfd.c preload/lib_i.c







































                                       - 17 -

            +--------------------------------------------------------------------+
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            |                                                                    |
            +--------------------------------------------------------------------+










                                       - 18 -


            88))  PPrreesseerrvviinngg  iinnffoorrmmaattiioonn oonn eexxeecc:: On successful execution
            of UNIX execve() system call, all file descriptors that  are
            not  explicitly  marked  as close-on-exec are left open, but
            the process'es image, including R-Unix state information, is
            completely  replaced  with another program. To allow the new
            program - it's instance of librunix.so - to  re-create  this
            information, R-Unix sets an environment variable RUNIX_STATE
            to a text representation of all  the  global  variables.  On
            initialization, librunix.so checks if RUNIX_STATE is set and
            if it is, just reconstructs global arrays and  recovers  RPC
            handle  from  rpc_fd instead of making another connection to
            the server.

            As  an  extra  precursion,  RUNIX_STATE  variable   includes
            current  process  pid  and  is  ignored  if it doesn't match
            getpid() result. This avoids using stale information.

            Source code: preload/lib_i_exec.c



            You are now left to your own fate.







































