# File subsystem

The file subsystem allows the system to interact with filesystems (FS) such as
FAT.

A FS is interacted with through the "fs" struct. It manages "file" structs which
are bound to that FS. What we call "files" are implementation-specific
structures that extend Stream [io/stream]. Handled in a generic manner, a file
is a stream.

In a FS, there are directories and files. Directories can contains files or
other directories. Both of them have a name. Both file and directories are
referred to through a 32-bit ID, uniquely identifying them.

Each FS has a root directory which is bound to ID 0. All the contents accessible
to that FS is contained in that root directory.

Each directory can be queried for a child (an element directly contained by that
directory) through a name lookup.

## The concept of FS IDs

When we refer to a file or directory ID, we refer to a single 32-bit number
uniquely identifying the file or directory in a particular FS. From this number,
we must be able to find the file or directory in the volume so that we can open
it or enumerate it. The ID of the root directory is always zero.

These IDs must be considered as unstable relative to changes to the FS. As long
as the hierarchy stays the same, IDs will be valid, but when it changes, IDs
might change. For example, if a file is removed from a directory, it's possible
that all IDs from that directory change too.

One should keep this limitation in mind when holding on to such an ID. For long
term storage of a FS reference, it's better to hold only a path in its string
form.

## bootfs and FS mappings

Although you can create as many FS structs as you want to access as many
FSes as you want, this unit provides the "bootfs" value which is bound to
the FS this system booted from. It's an important value because lookup for paths
that aren't prefixed with a map letter (see below) will happen in bootfs.

Although it's often harmless to change the bootfs value, it might cause
troubles, for example because of path references tied to [text/ed] buffers. But
the system itself doesn't generally keep FS references around, so unless you've
created those references yourself, go ahead, change bootfs as needed.

This unit also has a system to map FSes to case-insentive letters (from A to Z).
If a path begins with a letter followed by a ':', the lookup logic looks into
the mapping for a FS to perform its lookup into. Mappings are created through
the word "mapfs".

## Path finding logic

To make children lookup conveniently recursive, we have the concept of "path",
which is a hierarchical series of names. The "/" character is the path
separator.

For example, "foo/bar" refers to the "bar" child of the "foo" child of the
lookup base directory.

Path can contain a FS map prefix, for example "A:foo/bar", which will result in
the lookup of "foo/bar" to be performed on the FS mapped to the letter 'A'.

Such paths can be queried through "lookup ( name -- ?id fs-or-0 )" which takes
a path as a string and yields the ID of the found child as well as the FS in
which this child lives.

"lookup" always begins its lookup from the root of a FS.

There's also "lookuprel ( name dirid fs -- id-or-0 )" which can begin its lookup
from any specific subdirectory. These lookups can't contain FS map prefixes.

## File loading shortcuts

Feeding the Forth intepreter with the contents of a file is something you'll
want to do fairly often, which might not sit well with the verbosity of the FS
API. A shortcut to finding a path and then loading it is "f<<". This word,
followed by a path to load in the input stream, will call "interpretstream" on
that path.

It happens frequently that some piece of code depends on some other so we want
to load these files first. However, some units are depended upon by more than
one units and we don't want to load these pieces of code twice. To this end,
we have "needs" and "unit" (see [doc/usage/unit]).

"unit ..." creates an empty entry of the specified name in the dictionary.
"needs ..." checks if "..." exists in the dictionary. If it doesn't, we load the
file "<...>.fs". For example, if the unit lib/foo.fs depends on lib/bar.fs,
bar.fs will contain "unit lib/bar" and foo.fs will contain "needs lib/bar\nunit
lib/foo". This way, even if more than one unit have "needs lib/bar", lib/bar.fs
will only be loaded once.

## Booting filesystems

There's a bit of a chicken-and-egg problem at boot time. A Dusk system that
wants file capabilities (that is, a vast majority) will have to interpret
[io/stream.fs] and [fs/core.fs] right after [xcomp/boot.fs]. We don't have an
active bootfs yet because we have no FS implementation loaded. We don't even
have a means to access mass storage! However, filesystem abstractions need to
live in memory first, so we load them first.

Then, the system will load whatever units it needs to load for having a FS
implementation, through whatever means it can muster to do so (on most modern
systems, this means inlining the "payload code" right after the kernel code in
memory and interpret all this merry code from memory).

Until we do so, the FS words shouldn't actually be used. There's a special
provision for "needs" which is used in unit headers to specify dependencies.
During booting, when there is no FS yet it's the equivalent of the core version
of "needs", that is, it asserts that it exists and doesn't try to automatically
load it in memory.

Once it's done, once the boot process could manage to instantiate a concrete
Filesystem structure, it calls "bootfs! ( fs -- )" which then "activates" the
FS mechanism. FS-related words can then be used freely.

## Path literals

There are two words for path/file shortcuts: p" and f". These literals work both
in runtime and compile time mode. In compile time mode, it's not the FS ID that
is compiled, but the path string. In other words, path finding isn't done at
compile time but at runtime.

p"foo" is a shortcut for:

  "foo" lookup#

f"foo" is a shortcut for:

  p"foo" open

## A file grows when writing at the end of it

A file is a sized stream. When reading or writing to it, "pos" is updated so
that the next read or write takes place right after the previous operation
(unless a "seek" is used).

When reading up to a file's size, we don't read any longer, so "readbuf" yields
a zero sized range (that's EOF condition).

When writing, however, we try to grow the file by calling "resize". If it's
successful, write to the resized file. Otherwise, EOF condition.

## FS implementations don't seek their storage

It is an expected behavior for FS implementations to avoid seeking their
underlying storage, that is, changing their stream position. This behavior is
relied on, for example, in [xcomp/i386/pc/deploy] in the case where "install"
target's the bootfs storage (in other words, a self-update). If the FS seeks its
storage, data corruption occurs because the payload of "install" gets written
all over the place instead of where it should be.

To honor this promise, FS implementations should either use a [io/part] layer or
use [io/blk]'s "window" word instead of regular [io/stream] words on the
storage.

## API

### Struct

The FS struct encapsulate the logic for interacting with a particular file
system. It has many methods that need to be implemented by the extending struct
and supplies a sizeable set of convenience words on top of those methods.

struct FS
  field files \ 4b next, 4b "used?" followed by file struct
  field storage
  method child
  method info
  method openfile
  method children
  method newfile
  method newdir
  method remove
endstruct

files      Linked list of file structs tied to this filesystem.
storage    Stream [io/stream] structure upon which this filesystem is mounted.
           Some FS implementation might depend on a more specific stream such
           as [io/blk]

child ( name dirid fs -- id )
  Enumerate the contents of directory "dirid" and look for name "name". If
  found, returns the ID of the child (either a file or directory), otherwise,
  return 0. See below for the concept of "ID".

children ( id fs -- ... n )
  Given a parent id "id", yield an unordered list of the children it contains.
  "n" is the number of children and it's followed by that many number of FS IDs
  representing the children of the parent "id". The "." and ".." elements of a
  parent element (if present in the FS implementation) are never present in that
  list.

info ( id fs -- dir? size name )
  Yields "id"'s name, size in bytes and whether it's a directory. The "name"
  part is a string that might be in a static buffer and is only valid until the
  next "info" call.

openfile ( id file fs -- )
  Have the already allocated "file" struct target specified path id.

  Aborts on error.

newfile ( name dirid fs -- id )
  Create a new empty file named "name" in "dirid" and return the ID of the new
  file. Aborts on error.

newdir ( name dirid fs -- id )
  Create a new directory named "name" in "dirid" and return the ID of the new
  directory. Aborts on error.

remove ( id fs -- )
  Remove the element corresponding to "id" from the filesystem.

initfilestruct ( fs -- )
  Allocate, right "here", a new file structure that isn't tied to any file, but
  that must be ready to be passed to "openfile".

### Words

newfs ( <methods> tgtdrv -- fs )
  Allocate a new FS struct tied to the specified methods. You will typically not
  call this directly, as it's called by the FS implementations. "<methods>" is a
  list of pointers to method implementations for this FS, in reverse order
  ("initfilestruct" is the deepest, "child" is on top).

mapfs ( fs c -- )
  Map "fs" to prefix letter "c". Abort if "c" is not in the A-Z range.

pathfs ( path -- path fs )
  Yields the associated FS of a path. If "path" has a prefix letter, look it up
  in the mapping. If the mapping is set, yield that FS and trim the first 2
  letters from path. The new trimmed version of "path" lives in the string pool.
  If "path" has no prefix, fs=bootfs and "path" is left untouched.

?child ( name dirid fs -- id 1 OR name dirid fs 0 )
  Call "child" and possibly yield the resulting ID. If not found, PS is kept
  intact. This makes it convenient to subsequently call "newfile" or "newdir".

lookuprel ( path dirid fs -- id-or-0 )
  Recursively look for string "path" in directory "dirid" with the lookup logic
  decribed above in "Path finding logic".

lookup ( path -- ?id fs-or-0 )
  Call "lookuprel" with dirid=0 and fs=bootfs. If path is found, "id" is present
  and fs=bootfs. Otherwise, "id" is absent and fs=0.

  If "path" contains only a prefix, for example "A:", id=0 (root dir) and fs is
  the FS mapped to the prefix key.

lookup# ( path -- id fs )
  Same as lookup, but aborts with error if not found.

lookuprel# ( path dirid fs -- id )
  Same as lookuprel, but aborts with error if not found.

open ( id fs -- file )
  Open file targeting specified path ID, allocating or reusing a file struct,
  depending whether reuse is possible. Yields the file in question.

  Aborts on error.

openpath ( path -- file )
Stream* openpath(uchar *path);
  A shortcut for "lookup#" followed by "open" using the found ID.

ensurepath ( path dirid fs -- id f )
  Run a "lookuprel". If found, "id" is the result of "lookup" and f=1. If not,
  f=0 and "id" is the ID of the directory where such a path would live. If
  "path" has non-leaf (the last component of the path) elements that don't
  exist, a new directory is created for each of them.

  For example, "path/to/name" bootfs ensurepath would, on a bootfs that doesn't
  contain a "path" directory, create a "path" directory, then a "to" in it, then
  yield f=0 with "id" being the FSID of the "to" directory freshly created.

ensurepathrel ( dir? path dirid fs -- id fs )
  Calls "ensurepath" and, if it doesn't already exist, create a file or
  directory (depending on the "dir?" flag) at that path.
  Then, yield its "id fs" pair.

  If ensurepath yields f=1 but that the resulting id is of a different file/dir
  kind as the one requested, abort with error.

ensurefile ( path -- id fs )
  Shortcut for ensurepathrel with dir?=0 and pathfs lookup.

ensuredir ( path -- id fs )
  Shortcut for ensurepathrel with dir?=1 and pathfs lookup.

closeall ( fs -- )
  Close all files associated with this FS.

loadpath ( path -- )
  Lookup "path" then interpret is as Forth source. Automatically call "endunit"
  when the end of file is reached. Closes the file afterwards.

f<< ( "path" -- )
  Read path from input stream then call "loadpath" on bootfs.

exec<< ( w "path" -- )
  Find path then hijack the input stream so that it yields its contents, then
  call word "w". After "w" returns, close the opened file.
