# File subsystem

The file subsystem allows the system to interact with filesystems.

Files and directories all live within filesystems. More than one filesystem can
be active at once and each of these filesystems is designated by a letter. For
example, the boot filesystem is always "A". There's a maximum of 26 active
filesystems.

Filesystems contain files and directories. A file is a blob of data that exists
within a directory. A directory is a container for files and other directories.
Each filesystem has one root directory.

Files and directories are referred to with strings of characters that we call
"paths". Those paths begin with a filesystem identifier, followed by name of
directory and file. A directory name can be followed by a "/" character to
indicate that we want to refer to one of its children.

There is also the concept of an "active" directory. When paths don't start with
a drive letter, the path is relative to this active directory. A path starting
with "/" refers to the root directory of the active directory's filesystem. The
initial active directory is the boot filesystem's root directory (A:).

Examples:

A:                 root of filesystem A
A:foo.txt          file "foo.txt" inside A's root directory
B:foo/bar.fs       file "bar.fs" inside "foo" directory inside B's root
bar.fs             file "bar.fs" in active directory
/foo.fs            file "foo.fs" in active filesystem's root directory

You mount a filesystem through a struct that extends Filesystem. For example,
the FAT struct from fs/fat. Once that done, you will seldom have to interact
with that structure directly, but will rather do so through the File and Path
structs.

## The File struct

The File struct extends IO [doc/sys/io] and represents an opened file. You can
open a file through "Filesystem :open", but you'll typically go through Path,
which is more convenient.

From an operator perspective, the File is used pretty much like a regular IO, so
there's not much to say about this struct.

From a filesystem implementation perspective, the File base struct holds a
sector buffer for the file and implements almost all IO methods. The only one
it doesn't implement is ":resize". In turn, it defines its own methods which
need to be implemented by the FS-specific File implementation.

Fields:

used?  Whether this struct slot is currently used. If not, this area of memory
       can be used to open a new file.
fs     Filesystem struct that spawned this file.
pos    Current position of the file, in byte offset relative to the file's
       beginning.
size   Size in bytes of the file. Is updated when writing to the end of the file
       or when resizing.
secwin Sector buffer for this file. This is a SectorWindow struct and has a
       ":secwin" fstructbind.

Methods:

:place ( self -- ?a n-or-0 )
  Place the sector buffer so that it points to the current value of "pos". If
  "pos" doesn't have a corresponding sector on disk, n=0 and "a" is absent.
  Otherwise, read the appropriate sector into the buffer and yield the address
  representing "pos" as "a". "n" is nonzero and represents the number of bytes
  following that address that is a valid representation of the corresponding
  sector.

  For example, if the file's first data sector is 42, with a sector size of 512
  bytes and a "pos" of 123, then the 512 bytes of sector 42 are loaded into
  memory, let's say at address $1234. The result of :place will then be
  a=$1234+123 and n=512-123.

  The address thus yielded is expected to have a lifetime tied to the File
  struct. This means that it should be valid until the next :place call on that
  particular file.

Words:

:hold ( self -- )
  Set "used?" to 1.

:release ( self -- )
  Set "used?" to 0.

:new ( fs -- file )
  Create a new File struct tied to Filesystem "fs".

## Filesystem API

The Filesystem struct encapsulate the logic for interacting with a particular
file system. It has many methods that need to be implemented by the extending
struct and supplies a sizeable set of convenience words on top of those methods.

Fields:

drv        Drive pointer
flags      b0=writable
lastcursor Linked list of File structs tied to this filesystem.

Methods:

:child ( dirid name fs -- id )
  Enumerate the contents of directory "dirid" and look for name "name". If
  found, returns the ID of the child (either a file or directory), otherwise,
  return 0. See below for the concept of "ID".

:children ( id fs -- ... n )
  Given a parent id "id", yield an unordered list of the children it contains.
  "n" is the number of children and it's followed by that many number of FS IDs
  representing the children of the parent "id". The "." and ".." elements of a
  parent element are never present in that list.

:info ( id fs -- info )
  Returns a FSInfo structure (described below) corresponding to the specified
  ID. This structure is a static buffer and is only valid until the next :info
  call.

:open ( id fs -- file )
  Open file at path and return a handle (a "File" structure) through which other
  file-related word identify the target file. Once a file isn't used anymore, it
  should be closed with :close. Aborts on error.

:newfile ( dirid name fs -- id )
  Create a new empty file named "name" in "dirid" and return the ID of the new
  file. Aborts on error.

:newdir ( dirid name fs -- id )
  Create a new directory named "name" in "dirid" and return the ID of the new
  directory. Aborts on error.

:remove ( id fs -- )
  Remove the element corresponding to "id" from the filesystem.

:newcursor ( fs -- file )
  Allocates a new File struct and yield it.

Words:

:writeable? ( fs -- f )
  Is filesystem flagged as writeable?

:findfreecursor ( self -- file )
  Search in the "lastcursor" LL for an unused File struct. If it finds one,
  flag it as used and yield it. Otherwise, allocate a new File struct, add it
  to the LL, flag it as used and yield it.

:?newfile ( dirid name fs -- id )
  Create a new file if it doesn't exist. If it exists, return its ID.

:?newdir ( dirid name fs -- id )
  Create a new directory if it doesn't exist. If it exists, return its ID.

:copyfile ( dstid dstfs srcid srcfs -- )
  Copy file "srcid" residing in FS "srcfs" to existing file "dstid" residing in
  FS "dstfs".

:copydir ( dstid dstfs srcid srcfs -- )
  Recursively copy directory "srcid" residing in FS "srcfs" to existing
  directory "dstid" residing in FS "dstfs".

:new ( drv -- fs )
  Allocate a new Filesystem struct targeting drive "drv".

## FSInfo API

The structure returned by ":info" above goes as follows:

name -- a pointer to a string
size -- size of the file, or 0 for a directory
dir? -- whether the element is a directory

## SectorWindow

A SectorWindow is a buffer referencing one or more sectors and holding one
sector in memory. It manages the "dirty" status of the buffer and makes it easy
to seek within the window, automatically managing buffer status.

For example, one could open a SectorWindow of 4 512b sectors, starting at sector
$2a. The buffer would begin holding sector $2a in its :buf(. The user could
:seek relative address $212 within it, which would make it select sector $2b and
return address $12 relative to :buf(. If the dirty flag is set, the buffer
ensures that the contents is saved before loading a new one.

Fields:

drv     Target Drive
fsec    First sector of the window
cnt     Number of sectors in the window
sec     Sector currently in buffer
dirty   Whether the buffer is dirty

Words:

:new ( drv -- secwin )
  Create a new buffer targeting Drive "drv", on sector 0, cnt 1. Current buffer
  starts uninitialized.

:buf( ( self -- a )
  Address of the buffer.

:)buf ( self -- a )
  End of the buffer.

:move ( fsec cnt self -- )
  Move window to "fsec" with "cnt" sectors. This doesn't change or flush the
  current buffer.

:fulldrv ( self -- )
  Move the window so that it spans the whole drive.

:load ( sec self -- )
  Load sector "sec" in the buffer. Aborts if outside the window.

:flush ( self -- )
  If buffer is dirty, save it.

:seek ( pos self -- a? n-or-0 )
  Load, if needed, the correct sector so that the position "pos", in bytes
  relative to the beginning of "fsec", is available in the buffer. Returns its
  address in "a" as well as the number of bytes that can be read before the end
  of the buffer is reaches, as "n". If "pos" is outside the window, "n" is 0 and
  "a" is absent.

:next ( self -- a? n-or-0 )
  :seek the first byte of the sector following the one currently in the buffer.

:dirty! ( self -- )
  Make the buffer dirty.

## Path

The Path structure represents a path from any of the currently mounted
filesystems. It has 2 fields: "fs", which is a pointer to a Filesystem struct
and "id", which is the FS ID corresponding to the path. Path allocations are
managed through a small "rolling" buffer, which means that if you want to hold
on to a Path reference a little longer, you should copy it elsewhere.

:new ( fs id -- path )
  Allocate a new Path reference with the specified fields

:bi ( self -- id fs )
  Simply unpack "id fs" in one swift call.

:find ( str self -- path-or-0 )
  Find FS ID corresponding to path and return Path reference or 0 if not found.

:find# ( str self -- path )
  Like :find, but errors out if not found.

:fload ( self -- )
  Open specified path and interpret it as Forth source.

:floaded? ( self -- f )
  Returns whether this particular path has ever been loaded with :fload.

:chdir ( self -- )
  Change current directory to "self". In subsequent path finds, if the path
  doesn't start with "/", the search will start from this directory.

:iter ( self -- )
  Writes an iterator that iterates over all "self"'s children. During yields,
  "i" is a Path struct (which lives in RS during iteration) representing the
  path currently being iterated upon.

:open ( self -- file )
:info ( self -- info )
:child ( name self -- path-or-0 )
:children ( self -- ... n )
:copyfile ( dst self -- )
:copydir ( dst self -- )
  Convenience proxies to the corresponding methods in the Filesystem API.

:exec< ( w self -- )
  Open the path, call :exec< (see [doc/sys/io]) on the opened file, then close
  it.

## Global words

The file subsystem also supplies a set of global convenience words not tied to
any struct.

f<< ( "path" -- )
  Find path then interpret is as Forth source, then mark is as "loaded".

?f<< ( "path" -- )
  Find path then check if it's already marked as "loaded". If not, load it and
  mark it.

require ( "path" -- )
  Find path then check if it's already marked as "loaded". If not, abort with
  an error message.

exec<< ( w "path" -- )
  Find path then hijack the input stream so that it yields its contents, then
  call word "w". After "w" returns, close the opened file.

## Booting filesystems

There's a bit of a chicken-and-egg problem at boot time. A Dusk system that
wants file capabilities (that is, a vast majority) will have to interpret
[sys/io.fs] and [sys/file.fs] right after [xcomp/boot.fs]. We don't have an
active bootfs yet because we have no FS implementation loaded. We don't even
have a means to access mass storage! However, filesystem abstractions need to
live in memory first, so we load it first.

Then, the system will load whatever units it needs to load for having a FS
implementation, through whatever means it can muster to do so (on most modern
systems, this means inlining the "payload code" right after the kernel code in
memory and interpret all this merry code from memory).

Until we do so, the FS and Path words shouldn't actually be used. There's a
special provision for "?f<<" and "require" which are often used in unit headers
to specify dependencies. During booting, when there is no FS yet, they are
noops.

Once it's done, once the boot process could manage to instantiate a concrete
Filesystem structure, it calls "bootfs! ( fs -- )" which then "activates" the
FS mechanism. FS-related words can then be used freely.

After the boot process has activated the FS, it should call
"bootfloads, ( n --)" which is designed to generate fake "floaded," calls for
units loaded during the boot process so they aren't loaded twice. Here's an
example "finale" for a boot process:

  MyConcreteFS :new bootfs!
  2 bootfloads, /lib/somelib.fs /fs/concrete.fs

"/sys/io.fs" and "/sys/file.fs" must not be included in this list because they
are implicitly "fake loaded" by "bootfloads,".

## Global variables

The File subsystem has 2 important global variables:

bootfs is a structbind to the Filesystem structure from which the system was
booted. It's equivalent to the first (the "A" drive) filesystem.

curpath is a Path bind that points to the "current directory". It is initialized
to the root directory of bootfs but can be changed with Path :chdir.

## Paths

Paths uniquely identify a directory or a file on the current set of mounted
filesystems. Paths can take 3 forms: absolute, FS relative and relative.

Absolute paths begin with a drive letter ("A" is the boot drive, "B" is the
second one, etc.) followed by ":", followed by a relative path. Example:

a:lib/str.fs

FS relative are path relative to the currently "active" filesystem, that is, the
filesystem that contains the "current directory". Such paths begin by a "/"
followed by a relative path. Example:

/lib/str.fs

Relative paths are paths relative to the "current directory". These paths begin
directly with a name. Example for a "current directory" set to "/lib":

str.fs

## File ID design considerations

When we refer to a file or directory ID, we refer to a single 32-bit number
uniquely identifying the file or directory in the volume. From this number, we
must be able to find the file or directory in the volume so that we can open it
or enumerate it. The ID of the root directory is always zero.

That IDs are 32-bit make their manipulation straightforward. One might fear that
32 bits are not enough to identify all elements of big filesystems.

On many filesystems, 32 bits are enough. For example, in ext4, inodes always fit
32 bits, even in 64-bit mode.

For FAT32, it's another matter. The straightforward way to derive an ID from an
element is to return the offset of its DirEntry and divide it by 32. This means
that this ID scheme could only address 64GB volumes.

This would mean that the Filesystem API would be broken on a subset of
filesystems.  Does it mean that Dusk couldn't read them? No, only that file/dir
enumeration would have to go through FS-specific tools. I think that this
inconvenience is worth it if it means an overall simpler API.

## File loading shortcuts

Feeding the Forth intepreter with the contents of a file is something you'll
want to do fairly often, which might not sit well with the verbosity of Path's
API. A shortcut to finding a Path and then loading it is "f<<". This word,
followed by a path to load in the input stream, with call :fload on that path.

It happens frequently that some piece of code depends on some other so we want
to load these files first. However, some units are depended upon by more than
one units and we don't want to load these pieces of code twice. To this end,
whenever a file is loaded, we record its ID in a linked list to remember we
loaded it. If you use the "?f<<" word instead of "f<<", the specified file will
be loaded only if it wasn't already loaded.

Some units, such as subsystems [doc/dirs], are meant to be loaded explicitly by
the operator rather than implicitly as a transitive dependency. When a
particular unit depends on such units, it wants to asserts that its dependency
is loaded, not actually load it (if it's not loaded, something  is deeply wrong,
so abort). In these cases, you'll want to use "require" instead of "?f<<".

## Handle management and global structbinds

When we open a file, the filesystem returns a handle. The management of the
memory required to service these handles is specific to the filesystem
implementation, but we typically require significant memory space for each
handle for the contents buffer. Therefore, it's important to close handles when
we don't use them anymore so that this memory space can be used for another
file.

Handles opened directly by the system operator can be combersome to manage and
forgetting to close them is easy. To help with this, sys/file defines two global
structbinds, "file" and "dstfile" with accompanying helper words listed below.

The idea with those structbinds is that they're easily accessible to the sysop
and also implicitly close themselves when being overwritten. This way, the sysop
doesn't have to remember to close those handles as long as she use those
structbinds.

Also, those structbinds serve as a convenience. That is, some words will use
them to simplify their API (you don't have to pass them a handle, just have the
structbinds properly set).

Words that use "file" and "dstfile" are expected to never set them themselves.
They can seek, read, write to it, but the binding to the handles themselves is
the domain of the sysop.

file    Structbind to IO, defaults to 0
dstfile Structbind to IO, defaults to 0

: file$ ( -- )
  Reset "file" and "dstfile", closing them if necessary.

: >file ( hdl -- )
  Close current "file" and rebind it to "hdl".

: >dstfile ( hdl -- )
  Close current "dstfile" and rebind it to "hdl".

: file>dstfile ( -- )
  Close current "dstfile", set it to "file" and reset "file".

## Path literals

There are 2 words for Path/File shortcuts: p" and f". These literals work both
in runtime and compile time mode. In compile time mode, it's not the Path ID
that is compiled, but the path string. In other words, path finding isn't done
at compile time but at runtime.

p" foo" is a shortcut for:

  S" foo" curpath :find#

f" foo" is a shortcut for:

  p" foo" Path :open >file
