# File subsystem

Prerequisites: drive, sys/io

The file subsystem allows the system to interact with filesystems.

Files and directories all live within filesystems. More than one filesystem can
be active at once and each of these filesystems is designated by a letter. For
example, the boot filesystem is always "A". There's a maximum of 26 active
filesystems.

Filesystems contain files and directories. A file is a blob of data that exists
within a directory. A directory is a container for files and other directories.
Each filesystem has one root directory.

Files and directories are referred to with strings of characters that we call
"paths". Those paths begin with a filesystem identifier, followed by name of
directory and file. A directory name can be followed by a "/" character to
indicate that we want to refer to one of its children.

There is also the concept of an "active" directory. When paths don't start with
a drive letter, the path is relative to this active directory. A path starting
with "/" refers to the root directory of the active directory's filesystem. The
initial active directory is the boot filesystem's root directory (A:).

Examples:

A:                 root of filesystem A
A:foo.txt          file "foo.txt" inside A's root directory
B:foo/bar.fs       file "bar.fs" inside "foo" directory inside B's root
bar.fs             file "bar.fs" in active directory
/foo.fs            file "foo.fs" in active filesystem's root directory

## Filesystem API

When the code refers to a filesystem, it does so through a definite structure:

Fields:

drv    Drive pointer
flags  b0=writable

Methods:

:child ( dirid name fs -- id )
  Enumerate the contents of directory "dirid" and look for name "name". If
  found, returns the ID of the child (either a file or directory), otherwise,
  return 0. See below for the concept of "ID".

:info ( id fs -- info )
  Returns a FSInfo structure (described below) corresponding to the specified
  ID. This structure is a singleton and is only valid until the next :info call.

:open ( id fs -- hdl )
  Open file at path and return a handle (a "IO" structure) through which other
  file-related word identify the target file. Once a file isn't used anymore, it
  should be closed with :close. Aborts on error.

:iter ( dirid previd fs -- id-or-0 )
  Given a parent id "dirid", return the child id following "previd" in it. If
  previd is 0, return the first child. If no child follows "previd", return 0.
  The iteration process ignores "." and ".." elements.

:newfile ( dirid name fs -- id )
  Create a new empty file named "name" in "dirid" and return the ID of the new
  file. Aborts on error.

:newdir ( dirid name fs -- id )
  Create a new directory named "name" in "dirid" and return the ID of the new
  directory. Aborts on error.

... followed by whatever data the FS implementation needs.

In each method, "fs" refers to a filesystem, a pointer to the structure decribed
above.

Additionally, the Filesystem struct also has these convenience methods which are
independent of the underlying implementation:

:?newfile ( dirid name fs -- id )
  Create a new file if it doesn't exist. If it exists, return its ID.

:?newdir ( dirid name fs -- id )
  Create a new directory if it doesn't exist. If it exists, return its ID.

## FSInfo API

The structure returned by ":info" above goes as follows:

name -- a pointer to a string
size -- size of the file, or 0 for a directory
dir? -- whether the element is a directory

## Path API

The Path structure represents a path from any of the currently mounted
filesystems. It has 2 fields: "fs", which is a pointer to a Filesystem struct
and "id", which is the FS ID corresponding to the path. Path allocations are
managed through a small "rolling" buffer, which means that if you want to hold
on to a Path reference a little longer, you should copy it elsewhere.

:new ( fs id -- path )
  Allocate a new Path reference with the specified fields

:find ( str -- path-or-0 )
  Find FS ID corresponding to path and return Path reference or 0 if not found.

:find# ( str -- path )
  Like :find, but errors out if not found.

:fload ( self -- )
  Open specified path and interpret it as Forth source.

:floaded? ( self -- f )
  Returns whether this particular path has ever been loaded with :fload.

:chdir ( self -- )
  Change current directory to path. In subsequent path finds, if the path
  doesn't start with "/", the search will start from this directory.
  Aborts if path is not found.

:iter ( self -- )
  Writes an iterator that iterates over all "self"'s children. During yields,
  "i" is a Path struct (which lives in RS during iteration) representing the
  path currently being iterated upon.

:open ( self -- file )
:info ( self -- info )
:child ( name self -- path-or-0 )
  Convenience proxies to the corresponding methods in the Filesystem API.

:exec< ( w self -- )
  Open the path, call :exec< (see sys/io) on the opened file, then close it.

## Global words

f<< ( "path" -- )
  Find path then interpret is as Forth source, then mark is as "loaded"

?f<< ( "path" -- )
  Find path then check if it's already marked as "loaded" if not, load it and
  mark it.

exec<< ( w "path" -- )
  Find path then hijack the input stream so that it yields its contents, then
  call word "w". After "w" returns, close the opened file.

## Global variables

The File subsystem has 2 important global variables:

bootfs is a structbind to the Filesystem structure from which the system was
booted. It's used only until the sys/file subsystem is initialized.

curpath is a Path bind that points to the "current directory". It is initialized
to the root directory of bootfs but can be changed with Path :chdir.

## Paths

Paths uniquely identify a directory or a file on the current set of mounted
filesystems. Paths can take 3 forms: absolute, FS relative and relative.

Absolute paths begin with a drive letter ("A" is the boot drive, "B" is the
second one, etc.) followed by ":", followed by a relative path. Example:

a:lib/str.fs

FS relative are path relative to the currently "active" filesystem, that is, the
filesystem that contains the "current directory". Such paths begin by a "/"
followed by a relative path. Example:

/lib/str.fs

Relative paths are paths relative to the "current directory". These paths begin
directly with a name. Example for a "current directory" set to "/lib":

str.fs

## File ID design considerations

When we refer to a file or directory ID, we refer to a single 32-bit number
uniquely identifying the file or directory in the volume. From this number, we
must be able to find the file or directory in the volume so that we can open it
or enumerate it. The ID of the root directory is always zero.

That IDs are 32-bit make their manipulation straightforward. One might fear that
32 bits are not enough to identify all elements of big filesystems.

On many filesystems, 32 bits are enough. For example, in ext4, inodes always fit
32 bits, even in 64-bit mode.

For FAT32, it's another matter. The straightforward way to derive an ID from an
element is to return the offset of its DirEntry and divide it by 32. This means
that this ID scheme could only address 64GB volumes.

This would mean that the Filesystem API would be broken on a subset of
filesystems.  Does it mean that Dusk couldn't read them? No, only that file/dir
enumeration would have to go through FS-specific tools. I think that this
inconvenience is worth it if it means an overall simpler API.

## File loading shortcuts

Feeding the Forth intepreter with the contents of a file is something you'll
want to do fairly often, which might not sit well with the verbosity of Path's
API. A shortcut to finding a Path and then loading it is "f<<". This word,
followed by a path to load in the input stream, with call :fload on that path.

It happens frequently that some piece of code depends on some other so we want
to load these files first. However, some units are depended upon by more than
one units and we don't want to load these pieces of code twice. To this end,
whenever a file is loaded, we record its ID in a linked list to remember we
loaded it. If you use the "?f<<" word instead of "f<<", the specified file will
be loaded only if it wasn't already loaded.

Some units, such as subsystems [doc/dirs], are meant to be loaded explicitly by
the operator rather than implicitly as a transitive dependency. When a
particular unit depends on such units, it wants to asserts that its dependency
is loaded, not actually load it (if it's not loaded, something  is deeply wrong,
so abort). In these cases, you'll want to use "require" instead of "?f<<".

## Handle management and global structbinds

When we open a file, the filesystem returns a handle. The management of the
memory required to service these handles is specific to the filesystem
implementation, but we typically require significant memory space for each
handle for the contents buffer. Therefore, it's important to close handles when
we don't use them anymore so that this memory space can be used for another
file.

Handles opened directly by the system operator can be combersome to manage and
forgetting to close them is easy. To help with this, sys/file defines two global
structbinds, "file" and "dstfile" with accompanying helper words listed below.

The idea with those structbinds is that they're easily accessible to the sysop
and also implicitly close themselves when being overwritten. This way, the sysop
doesn't have to remember to close those handles as long as she use those
structbinds.

Also, those structbinds serve as a convenience. That is, some words will use
them to simplify their API (you don't have to pass them a handle, just have the
structbinds properly set).

Words that use "file" and "dstfile" are expected to never set them themselves.
They can seek, read, write to it, but the binding to the handles themselves is
the domain of the sysop.

file    Structbind to IO, defaults to 0
dstfile Structbind to IO, defaults to 0

: file$ ( -- )
  Reset "file" and "dstfile", closing them if necessary.

: >file ( hdl -- )
  Close current "file" and rebind it to "hdl".

: >dstfile ( hdl -- )
  Close current "dstfile" and rebind it to "hdl".

: file>dstfile ( -- )
  Close current "dstfile", set it to "file" and reset "file".

## Path literals

There are 2 words for Path/File shortcuts: p" and f". These literals work both
in runtime and compile time mode. In compile time mode, it's not the Path ID
that is compiled, but the path string. In other words, path finding isn't done
at compile time but at runtime.

p" foo" is a shortcut for:

  S" foo" curpath :find#

f" foo" is a shortcut for:

  p" foo" Path :open >file
