# File subsystem

The file subsystem allows the system to interact with filesystems (FS) such as
FAT.

A FS is interacted with through the "fs" struct. It manages "file" structs which
are bound to that FS. File structs extend the IO struct [io/stream].

In a FS, there are directories and files. Directories can contains files or
other directories. Both of them have a name. Both file and directories are
referred to through a 32-bit ID, uniquely identifying them.

Each FS has a root directory which is bound to ID 0. All the contents accessible
to that FS is contained in that root directory.

Each directory can be queried for a child (an element directly contained by that
directory) through a name lookup.

## The concept of FS IDs

When we refer to a file or directory ID, we refer to a single 32-bit number
uniquely identifying the file or directory in a particular FS. From this number,
we must be able to find the file or directory in the volume so that we can open
it or enumerate it. The ID of the root directory is always zero.

These IDs must be considered as unstable relative to changes to the FS. As long
as the hierarchy stays the same, IDs will be valid, but when it changes, IDs
might change. For example, if a file is removed from a directory, it's possible
that all IDs from that directory change too.

One should keep this limitation in mind when holding on to such an ID. For long
term storage of a FS reference, it's better to hold only a path in its string
form.

## bootfs and FS mappings

Although you can create as many FS structs as you want to access as many
FSes as you want, this unit provides the "bootfs" value which is bound to
the FS this system booted from. It's an important value because lookup for paths
that aren't prefixed with a map letter (see below) will happen in bootfs.

Although it's often harmless to change the bootfs value, it might cause
troubles, for example because of path references tied to [text/ed] buffers. But
the system itself doesn't generally keep FS references around, so unless you've
created those references yourself, go ahead, change bootfs as needed.

This unit also has a system to map FSes to case-insentive letters (from A to Z).
If a path begins with a letter followed by a ':', the lookup logic looks into
the mapping for a FS to perform its lookup into. Mappings are created through
the word "mapfs".

## Path finding logic

To make children lookup conveniently recursive, we have the concept of "path",
which is a hierarchical series of names. The "/" character is the path
separator.

For example, "foo/bar" refers to the "bar" child of the "foo" child of the
lookup base directory.

Path can contain a FS map prefix, for example "A:foo/bar", which will result in
the lookup of "foo/bar" to be performed on the FS mapped to the letter 'A'.

Such paths can be queried through "lookup ( name -- ?id fs-or-0 )" which takes
a path as a string and yields the ID of the found child as well as the FS in
which this child lives.

"lookup" always begins its lookup from the root of a FS.

There's also "lookuprel ( name dirid fs -- id-or-0 )" which can begin its lookup
from any specific subdirectory. These lookups can't contain FS map prefixes.

## File loading shortcuts

Feeding the Forth intepreter with the contents of a file is something you'll
want to do fairly often, which might not sit well with the verbosity of the FS
API. A shortcut to finding a path and then loading it is "f<<". This word,
followed by a path to load in the input stream, will call "interpretstream" on
that path.

It happens frequently that some piece of code depends on some other so we want
to load these files first. However, some units are depended upon by more than
one units and we don't want to load these pieces of code twice. To this end,
we have "needs" and "unit" (see [doc/usage/unit]).

"unit ..." creates an empty entry of the specified name in the dictionary.
"needs ..." checks if "..." exists in the dictionary. If it doesn't, we load the
file "<...>.fs". For example, if the unit lib/foo.fs depends on lib/bar.fs,
bar.fs will contain "unit lib/bar" and foo.fs will contain "needs lib/bar\nunit
lib/foo". This way, even if more than one unit have "needs lib/bar", lib/bar.fs
will only be loaded once.

## Booting filesystems

There's a bit of a chicken-and-egg problem at boot time. A Dusk system that
wants file capabilities (that is, a vast majority) will have to interpret
[io/stream.fs] and [fs/core.fs] right after [xcomp/boot.fs]. We don't have an
active bootfs yet because we have no FS implementation loaded. We don't even
have a means to access mass storage! However, filesystem abstractions need to
live in memory first, so we load them first.

Then, the system will load whatever units it needs to load for having a FS
implementation, through whatever means it can muster to do so (on most modern
systems, this means inlining the "payload code" right after the kernel code in
memory and interpret all this merry code from memory).

Until we do so, the FS words shouldn't actually be used. There's a special
provision for "needs" which is used in unit headers to specify dependencies.
During booting, when there is no FS yet it's the equivalent of the core version
of "needs", that is, it asserts that it exists and doesn't try to automatically
load it in memory.

Once it's done, once the boot process could manage to instantiate a concrete
Filesystem structure, it calls "bootfs! ( fs -- )" which then "activates" the
FS mechanism. FS-related words can then be used freely.

## Path literals

There are two words for path/file shortcuts: p" and f". These literals work both
in runtime and compile time mode. In compile time mode, it's not the FS ID that
is compiled, but the path string. In other words, path finding isn't done at
compile time but at runtime.

p"foo" is a shortcut for:

  "foo" lookup#

f"foo" is a shortcut for:

  p"foo" open

## File API

A File is basically a stream [io/stream] with a position and size. While its
structure is in big part used as part of a filesystem, a File struct can be used
independently of a FS, for example in [fs/memfile] or [mem/pool].

When used as part of an FS, you'll typically get a File reference through the
"open" word.

To allow file cursors to be reused, IO's "close" method needs to be called.

### Struct

extends Stream struct File
  fields used? pos size
  method place  ( file -- ?a n-or-0 )
  method dirty! ( file -- )
  method resize ( sz file -- )
  method flush  ( file -- )
endstruct

used?  Whether this struct slot is currently used. If not, this area of memory
       can be used to open a new file.
pos    Current position of the file, in byte offset relative to the file's
       beginning.
size   Size in bytes of the file. Is updated when writing to the end of the file
       or when resizing.

place ( file -- ?a n-or-0 )
  Place the sector buffer so that it points to the current value of "pos". If
  "pos" doesn't have a corresponding sector on disk, n=0 and "a" is absent.
  Otherwise, read the appropriate sector into the buffer and yield the address
  representing "pos" as "a". "n" is nonzero and represents the number of bytes
  following that address that is a valid representation of the corresponding
  sector.

  For example, if the file's first data sector is 42, with a sector size of 512
  bytes and a "pos" of 123, then the 512 bytes of sector 42 are loaded into
  memory, let's say at address $1234. The result of :place will then be
  a=$1234+123 and n=512-123.

  The address thus yielded is expected to have a lifetime tied to the File
  struct. This means that it should be valid until the next :place call on that
  particular file.

dirty! ( file -- )
  Set current buffer as dirty, that is, in need of sync with mass storage.

resize ( n file -- )
  Change the size of target I/O structure, in bytes.

flush ( file -- )
  If the file's buffer is "dirty" (has been changed compared to the permanent
  storage it represents), save that data to permanent storage and flag the
  handle as clean.

### Words

seek ( n file -- )
  Change current position to "n", which is a byte offset.

truncate ( file -- )
  Truncate the I/O to current position, inclusively. That is, a read operation
  directly following a truncate will always be EOF.

maxn ( file -- n )
  Yields the number of bytes between "pos" and the end of the file.

newfilestruct ( 'place -- file )
  Create a new file struct with the specified "place" method.

## FS API

The FS struct encapsulate the logic for interacting with a particular file
system. It has many methods that need to be implemented by the extending struct
and supplies a sizeable set of convenience words on top of those methods.

### Struct

struct FS
  field files \ 4b next followed by file struct
  field tgtdrv
  method child
  method info
  method open
  method children
  method newfile
  method newdir
  method remove
endstruct

files      Linked list of file structs tied to this filesystem.
tgtdrv     Drive [io/drive] structure upon which this filesystem is mounted.

child ( name dirid fs -- id )
  Enumerate the contents of directory "dirid" and look for name "name". If
  found, returns the ID of the child (either a file or directory), otherwise,
  return 0. See below for the concept of "ID".

children ( id fs -- ... n )
  Given a parent id "id", yield an unordered list of the children it contains.
  "n" is the number of children and it's followed by that many number of FS IDs
  representing the children of the parent "id". The "." and ".." elements of a
  parent element (if present in the FS implementation) are never present in that
  list.

info ( id fs -- dir? size name )
  Yields "id"'s name, size in bytes and whether it's a directory. The "name"
  part is a string that might be in a static buffer and is only valid until the
  next "info" call.

open ( id fs -- file )
  Open file at path and return a file through which other file-related word
  identify the target file. Once a file isn't used anymore, it should be closed
  with close. Aborts on error.

newfile ( name dirid fs -- id )
  Create a new empty file named "name" in "dirid" and return the ID of the new
  file. Aborts on error.

newdir ( name dirid fs -- id )
  Create a new directory named "name" in "dirid" and return the ID of the new
  directory. Aborts on error.

remove ( id fs -- )
  Remove the element corresponding to "id" from the filesystem.

### Words

mapfs ( fs c -- )
  Map "fs" to prefix letter "c". Abort if "c" is not in the A-Z range.

pathfs ( path -- path fs )
  Yields the associated FS of a path. If "path" has a prefix letter, look it up
  in the mapping. If the mapping is set, yield that FS and trim the first 2
  letters from path. The new trimmed version of "path" lives in the string pool.
  If "path" has no prefix, fs=bootfs and "path" is left untouched.

?reusefile ( fs -- ?file f )
  Search in the "files" LL for an unused file. If it finds one, f=1, "file" is
  present and it's "used?" flag is set to 1.
  Otherwise, f=0" and a new "files" entry is created, pointing to here.
  Therefore, any call to this word must be followed by a conditional creation of
  a file struct right here on the spot.

?child ( name dirid fs -- id 1 OR name dirid fs 0 )
  Call "child" and possibly yield the resulting ID. If not found, PS is kept
  intact. This makes it convenient to subsequently call "newfile" or "newdir".

lookuprel ( path dirid fs -- id-or-0 )
  Recursively look for string "path" in directory "dirid" with the lookup logic
  decribed above in "Path finding logic".

lookup ( path -- ?id fs-or-0 )
  Call "lookuprel" with dirid=0 and fs=bootfs. If path is found, "id" is present
  and fs=bootfs. Otherwise, "id" is absent and fs=0.

  If "path" contains only a prefix, for example "A:", id=0 (root dir) and fs is
  the FS mapped to the prefix key.

lookup# ( path -- id fs )
  Same as lookup, but aborts with error if not found.

lookuprel# ( path dirid fs -- id )
  Same as lookuprel, but aborts with error if not found.

openpath ( path -- file )
  A shortcut for "lookup#" followed by "open" using the found ID.

ensurepath ( dirid path fs -- id f )
  Run a "lookuprel". If found, "id" is the result of "lookup" and f=1. If not,
  f=0 and "id" is the ID of the directory where such a path would live. If
  "path" has non-leaf (the last component of the path) elements that don't
  exist, a new directory is created for each of them.

  For example, "path/to/name" bootfs ensurepath would, on a bootfs that doesn't
  contain a "path" directory, create a "path" directory, then a "to" in it, then
  yield f=0 with "id" being the FSID of the "to" directory freshly created.

ensurefile ( path -- id fs )
  Calls "ensurepath" and, if it doesn't already exist, create a file at that
  path. Then, yield its "id fs" pair.

ensuredir ( path -- id fs )
  Same as "ensurefile", but creates a directory instead.

copyfile ( dstid dstfs srcid srcfs -- )
  Copy file "srcid" residing in FS "srcfs" to existing file "dstid" residing in
  FS "dstfs".

copydir ( dstid dstfs srcid srcfs -- )
  Recursively copy directory "srcid" residing in FS "srcfs" to existing
  directory "dstid" residing in FS "dstfs".

  When ran on a destination that already exists, it creates new files/dirs as
  needed and when a file already exists, it is overwritten.

closeall ( fs -- )
  Close all files associated with this FS.

newfs ( <methods> tgtdrv -- fs )
  Allocate a new FS struct tied to the specified methods. You will typically not
  call this directly, as it's called by the FS implementations. "<methods>" is a
  list of pointers to method implementations for this FS, in reverse order
  ("remove" is the deepest, "child" is on top).

listdir ( dirid fs -- )
  Print all children of a given directory, one per line. Children can be files
  and/or subdirectories.

loadpath ( path -- )
  Lookup "path" then interpret is as Forth source. Automatically call "endunit"
  when the end of file is reached. Closes the file afterwards.

f<< ( "path" -- )
  Read path from input stream then call "loadpath" on bootfs.

exec<< ( w "path" -- )
  Find path then hijack the input stream so that it yields its contents, then
  call word "w". After "w" returns, close the opened file.
