# File subsystem

The file subsystem allows the system to interact with filesystems (FS) such as
FAT.

A FS is interacted with through the "fs" struct, which sits on top of a drive
struct, defined in this unit. It manages "file" structs which are bound to that
FS. File structs extend the IO struct [doc/io/stream].

In a FS, there are directories and files. Directories can contains files or
other directories. Both of them have a name. Both file and directories are
referred to through a 32-bit ID, uniquely identifying them.

Each FS has a root directory which is bound to ID 0. All the contents accessible
to that FS is contained in that root directory.

Each directory can be queried for a child (an element directly contained by that
directory) through a name lookup.

Although you can create as many FS structs as you want to access as many
FSes as you want, this unit provides the "bootfs" value which is bound to
the FS this system booted from. It's an important value because the whole
convenience layer ("f<<", "needs", etc.) is bound to it.

## The concept of FS IDs

When we refer to a file or directory ID, we refer to a single 32-bit number
uniquely identifying the file or directory in the volume. From this number, we
must be able to find the file or directory in the volume so that we can open it
or enumerate it. The ID of the root directory is always zero.

These IDs must be considered as unstable relative to changes to the FS. As long
as the hierarchy stays the same, IDs will be valid, but when it changes, IDs
might change. For example, if a file is removed from a directory, it's possible
that all IDs from that directory change too.

One should keep this limitation in mind when holding on to such an ID. For long
term storage of a FS reference, it's better to hold only a path in its string
form.

## Path finding logic

To make children lookup conveniently recursive, we have the concept of "path",
which is a hierarchical series of names. The "/" character is the path
separator.

For example, "foo/bar" refers to the "bar" child of the "foo" child of the
lookup base directory.

Such paths can be queried through "lookup ( name fs -- id-or-0 )" which takes
a path as a string and yields the ID of the found child or 0 if it doesn't
exist.

"lookup" has the root directory as its lookup base directory.

"lookuprel" begins its lookup from an arbitrary directory.

## File loading shortcuts

Feeding the Forth intepreter with the contents of a file is something you'll
want to do fairly often, which might not sit well with the verbosity of the FS
API. A shortcut to finding a path and then loading it is "f<<". This word,
followed by a path to load in the input stream, will call "IO :interpret" on
that path.

It happens frequently that some piece of code depends on some other so we want
to load these files first. However, some units are depended upon by more than
one units and we don't want to load these pieces of code twice. To this end,
we have "needs" and "unit" (see [doc/usage/unit]).

"unit ..." creates an empty entry of the specified name in the dictionary.
"needs ..." checks if "..." exists in the dictionary. If it doesn't, we load the
file "/<...>.fs".  For example, if the unit /lib/foo.fs depends on /lib/bar.fs,
bar.fs will contain "unit lib/bar" and foo.fs will contain
"needs lib/bar\nunit lib/foo". This way, even if more than one unit have
"needs lib/bar", /lib/bar.fs will only be loaded once.

## Booting filesystems

There's a bit of a chicken-and-egg problem at boot time. A Dusk system that
wants file capabilities (that is, a vast majority) will have to interpret
[io/stream.fs] and [fs/core.fs] right after [xcomp/boot.fs]. We don't have an
active bootfs yet because we have no FS implementation loaded. We don't even
have a means to access mass storage! However, filesystem abstractions need to
live in memory first, so we load them first.

Then, the system will load whatever units it needs to load for having a FS
implementation, through whatever means it can muster to do so (on most modern
systems, this means inlining the "payload code" right after the kernel code in
memory and interpret all this merry code from memory).

Until we do so, the FS words shouldn't actually be used. There's a special
provision for "needs" which is used in unit headers to specify dependencies.
During booting, when there is no FS yet it's the equivalent of the core version
of "needs", that is, it asserts that it exists and doesn't try to automatically
load it in memory.

Once it's done, once the boot process could manage to instantiate a concrete
Filesystem structure, it calls "bootfs! ( fs -- )" which then "activates" the
FS mechanism. FS-related words can then be used freely.

## Path literals

There are two words for path/file shortcuts: p" and f". These literals work both
in runtime and compile time mode. In compile time mode, it's not the FS ID that
is compiled, but the path string. In other words, path finding isn't done at
compile time but at runtime.

p"foo" is a shortcut for:

  "foo" bootfs lookup#

f"foo" is a shortcut for:

  p"foo" bootfs open

## File API

The file struct extends IO [io/stream] and represents an opened file. You can
open a file through "open".

A file is a seekable IO that has a size.

To allow file cursors to be reused, IO's "close" method needs to be called.

### Constants

FILESZ  Size in bytes of a file struct

### Fields

used?  Whether this struct slot is currently used. If not, this area of memory
       can be used to open a new file.
pos    Current position of the file, in byte offset relative to the file's
       beginning.
size   Size in bytes of the file. Is updated when writing to the end of the file
       or when resizing.

### Methods

place ( file -- ?a n-or-0 )
  Place the sector buffer so that it points to the current value of "pos". If
  "pos" doesn't have a corresponding sector on disk, n=0 and "a" is absent.
  Otherwise, read the appropriate sector into the buffer and yield the address
  representing "pos" as "a". "n" is nonzero and represents the number of bytes
  following that address that is a valid representation of the corresponding
  sector.

  For example, if the file's first data sector is 42, with a sector size of 512
  bytes and a "pos" of 123, then the 512 bytes of sector 42 are loaded into
  memory, let's say at address $1234. The result of :place will then be
  a=$1234+123 and n=512-123.

  The address thus yielded is expected to have a lifetime tied to the File
  struct. This means that it should be valid until the next :place call on that
  particular file.

dirty! ( file -- )
  Set current buffer as dirty, that is, in need of sync with mass storage.

resize ( n file -- )
  Change the size of target I/O structure, in bytes.

flush ( file -- )
  If the file's buffer is "dirty" (has been changed compared to the permanent
  storage it represents), save that data to permanent storage and flag the
  handle as clean.

### Words

seek ( n file -- )
  Change current position to "n", which is a byte offset.

truncate ( file -- )
  Truncate the I/O to current position, inclusively. That is, a read operation
  directly following a :truncate will always be EOF.

maxn ( file -- n )
  Yields the number of bytes between "pos" and the end of the file.

newfilestruct ( 'place -- file )
  Create a new file struct with the specified "place" method.

## FS API

The FS struct encapsulate the logic for interacting with a particular file
system. It has many methods that need to be implemented by the extending struct
and supplies a sizeable set of convenience words on top of those methods.

### Constants

FSSZ  Size in bytes of a FS struct

### Fields

files      Linked list of file structs tied to this filesystem.

### Methods

child ( dirid name fs -- id )
  Enumerate the contents of directory "dirid" and look for name "name". If
  found, returns the ID of the child (either a file or directory), otherwise,
  return 0. See below for the concept of "ID".

children ( id fs -- ... n )
  Given a parent id "id", yield an unordered list of the children it contains.
  "n" is the number of children and it's followed by that many number of FS IDs
  representing the children of the parent "id". The "." and ".." elements of a
  parent element (if present in the FS implementation) are never present in that
  list.

info ( id fs -- dir? size name )
  Yields "id"'s name, size in bytes and whether it's a directory. The "name"
  part is a string that might be in a static buffer and is only valid until the
  next "info" call.

open ( id fs -- file )
  Open file at path and return a file through which other file-related word
  identify the target file. Once a file isn't used anymore, it should be closed
  with close. Aborts on error.

newfile ( dirid name fs -- id )
  Create a new empty file named "name" in "dirid" and return the ID of the new
  file. Aborts on error.

newdir ( dirid name fs -- id )
  Create a new directory named "name" in "dirid" and return the ID of the new
  directory. Aborts on error.

remove ( id fs -- )
  Remove the element corresponding to "id" from the filesystem.

### Words

?reusefile ( fs -- ?file f )
  Search in the "files" LL for an unused file. If it finds one, f=1, "file" is
  present and it's "used?" flag is set to 1.
  Otherwise, f=0" and a new "files" entry is created, pointing to here.
  Therefore, any call to this word must be followed by a conditional creation of
  a file struct right here on the spot.

?child ( dirid name fs -- id 1 OR dirid name fs 0 )
  Call "child" and possibly yield the resulting ID. If not found, PS is kept
  intact. This makes it convenient to subsequently call "newfile" or "newdir".

lookuprel ( dirid path fs -- id-or-0 )
  Recursively look for string "path" in directory "dirid" with the lookup logic
  decribed above in "Path finding logic".

lookup ( path fs -- id-or-0 )
  Call "lookuprel" with dirid=0.

lookup# ( path fs -- id )
  Same as lookup, but aborts with error if not found.

lookuprel# ( dirid path fs -- id )
  Same as lookuprel, but aborts with error if not found.

?lookup ( path fs -- id f )
  Lookup for "path" in "fs". If found, "id" is the result of "lookup" and f=1.
  If not, f=0 and "id" is the ID of the directory where such a path would live.
  We assume that lookup succeeds up to the "leaf" part of "path". If it doesn't,
  we abort with "directory not found".

ensurepath ( path fs -- id f )
  Lookup for "path" in "fs". If found, "id" is the result of "lookup" and f=1.
  If not, f=0 and "id" is the ID of the directory where such a path would live.
  If "path" has non-leaf (the last component of the path) elements that don't
  exist, a new directory is created for each of them.

  For example, "path/to/name" bootfs ensurepath would, on a bootfs that doesn't
  contain a "path" directory, create a "path" directory, then a "to" in it, then
  yield f=0 with "id" being the FSID of the "to" directory freshly created.

openpath ( strpath fs -- file )
  A shortcut for "lookup#" followed by "open" using the found ID.

?newfile ( path fs -- id )
  If "path" exists and is a file simply yield its FSID. If it exists but is a
  directory, abort with error.

  If it doesn't exist, ensure that the directory chain leading to the last
  component of the path exists, then create the file corresponding to the last
  component of the path, then yield the FSID of the created file.

?newdir ( path fs -- id )
  Same as "?newfile", but creates a directory instead.

copyfile ( dstid dstfs srcid srcfs -- )
  Copy file "srcid" residing in FS "srcfs" to existing file "dstid" residing in
  FS "dstfs".

copydir ( dstid dstfs srcid srcfs -- )
  Recursively copy directory "srcid" residing in FS "srcfs" to existing
  directory "dstid" residing in FS "dstfs".

closeall ( fs -- )
  Close all files associated with this FS.

newfs ( <methods> -- fs )
  Allocate a new FS struct tied to the specified methods. You will typically not
  call this directly, as it's called by the FS implementations. "<methods>" is a
  list of pointers to method implementations for this FS, in reverse order
  ("remove" is the deepest, "child" is on top).

loadpath ( path fs -- )
  Lookup "path" then interpret is as Forth source. Automatically call "endunit"
  when the end of file is reached. Closes the file afterwards.

f<< ( "path" -- )
  Read path from input stream then call "loadpath" on bootfs.

exec<< ( w "path" -- )
  Find path then hijack the input stream so that it yields its contents, then
  call word "w". After "w" returns, close the opened file.
