# File subsystem

The file subsystem allows the system to interact with filesystems (FS) such as
FAT.

A FS is interacted with through the Filesystem struct, which sits on top of a
Drive struct, defined in this unit. It manages "cursors", that is, instances of
the File struct bound to that FS. File structs extend the IO struct
[doc/sys/io].

In a FS, there are directories and files. Directories can contains files or
other directories. Both of them have a name. Both file and directories are
referred to through a 32-bit ID, uniquely identifying them.

Each FS has a root directory which is bound to ID 0. All the contents accessible
to that FS is contained in that root directory.

Each directory can be queried for a child (an element directly contained by that
directory) through a name lookup.

Although you can create as many Filesystem structs as you want to access as many
FSes as you want, this unit provides the "bootfs" bind{} which is bound to
the FS this system booted from. It's an important bind{} because the whole
convenience layer (for example, the "f<<" word) is bound to it.

While you can load code from FSes other than "bootfs" by using IO methods, one
thing you cannot do (at least, not without a major headache) is to have the
"floaded" linked list, which keeps track of the loaded units, refer to those
other FSes.

This means that, in general, Dusk can only load units from its boot FS.

## The concept of FS IDs

When we refer to a file or directory ID, we refer to a single 32-bit number
uniquely identifying the file or directory in the volume. From this number, we
must be able to find the file or directory in the volume so that we can open it
or enumerate it. The ID of the root directory is always zero.

That IDs are 32-bit make their manipulation straightforward. One might fear that
32 bits are not enough to identify all elements of big filesystems.

On many filesystems, 32 bits are enough. For example, in ext4, inodes always fit
32 bits, even in 64-bit mode.

For FAT32, it's another matter. The straightforward way to derive an ID from an
element is to return the offset of its DirEntry and divide it by 32. This means
that this ID scheme could only address 64GB volumes.

This would mean that the Filesystem API would be broken on a subset of
filesystems. Does it mean that Dusk couldn't read them? No, only that file/dir
enumeration would have to go through FS-specific tools. I think that this
inconvenience is worth it if it means an overall simpler API.

## Path finding logic

To make children lookup conveniently recursive, we have the concept of "path",
which is a hierarchical series of names. The "/" character is the path
separator.

For example, "/foo/bar" refers to the "bar" child of the "foo" child of the root
directory. In that example, the leading "/" means that the lookup starts from
root.

A path can be relative, for example "foo/bar". In that case, the lookup starts
from another directory, specified through the API.

Such paths can be queried through "Filesystem :find ( name self -- id-or-0 )"
which takes a path as a string and yields the ID of the found child or 0 if it
doesn't exist.

":find" begins its search at "Filesystem curdir" field, which defaults to 0
(root).

## Booting filesystems

There's a bit of a chicken-and-egg problem at boot time. A Dusk system that
wants file capabilities (that is, a vast majority) will have to interpret
[sys/io.fs] and [sys/file.fs] right after [xcomp/boot.fs]. We don't have an
active bootfs yet because we have no FS implementation loaded. We don't even
have a means to access mass storage! However, filesystem abstractions need to
live in memory first, so we load it first.

Then, the system will load whatever units it needs to load for having a FS
implementation, through whatever means it can muster to do so (on most modern
systems, this means inlining the "payload code" right after the kernel code in
memory and interpret all this merry code from memory).

Until we do so, the FS words shouldn't actually be used. There's a special
provision for "?f<<" and "require" which are often used in unit headers to
specify dependencies. During booting, when there is no FS yet, they are noops.

Once it's done, once the boot process could manage to instantiate a concrete
Filesystem structure, it calls "bootfs! ( fs -- )" which then "activates" the
FS mechanism. FS-related words can then be used freely.

After the boot process has activated the FS, it should call
"bootfloads, ( n --)" which is designed to generate fake "floaded," calls for
units loaded during the boot process so they aren't loaded twice. Here's an
example "finale" for a boot process:

  MyConcreteFS :new bootfs!
  2 bootfloads, /lib/somelib.fs /fs/concrete.fs

"/sys/io.fs" and "/sys/file.fs" must not be included in this list because they
are implicitly "fake loaded" by "bootfloads,".

## File loading shortcuts

Feeding the Forth intepreter with the contents of a file is something you'll
want to do fairly often, which might not sit well with the verbosity of the FS
API. A shortcut to finding a path and then loading it is "f<<". This word,
followed by a path to load in the input stream, will call "IO :interpret" on
that path.

It happens frequently that some piece of code depends on some other so we want
to load these files first. However, some units are depended upon by more than
one units and we don't want to load these pieces of code twice. To this end,
whenever a file is loaded, we record its ID in a linked list to remember we
loaded it. If you use the "?f<<" word instead of "f<<", the specified file will
be loaded only if it wasn't already loaded.

Some units, such as subsystems [doc/dirs], are meant to be loaded explicitly by
the operator rather than implicitly as a transitive dependency. When a
particular unit depends on such units, it wants to asserts that its dependency
is loaded, not actually load it (if it's not loaded, something is deeply wrong,
so abort). In these cases, you'll want to use "require" instead of "?f<<".

### Relative paths during file loading

The "f<<" and "?f<<" words use "Filesystem :exec<" for interpreting the file.
Because that word changes "curdir" to "finddir" while this occurs, this allows
us to use path relative to the loaded unit in that very unit.

For example, if you load "/foo/bar.fs" and that it contains "?f<< baz.fs" in it,
it's the file "/foo/baz.fs" that is going to be loaded.

## Handle management and global bind{}s

When we open a file, the filesystem returns a handle. The management of the
memory required to service these handles is specific to the filesystem
implementation, but we typically require significant memory space for each
handle for the contents buffer. Therefore, it's important to close handles when
we don't use them anymore so that this memory space can be used for another
file.

Handles opened directly by the system operator can be combersome to manage and
forgetting to close them is easy. To help with this, sys/file defines two global
bind{}s, "file" and "dstfile" with accompanying helper words listed below.

The idea with those bind{}s is that they're easily accessible to the sysop
and also implicitly close themselves when being overwritten. This way, the sysop
doesn't have to remember to close those handles as long as she use those
bind{}s.

Also, those bind{}s serve as a convenience. That is, some words will use
them to simplify their API (you don't have to pass them a handle, just have the
bind{}s properly set).

Words that use "file" and "dstfile" are expected to never set them themselves.
They can seek, read, write to it, but the binding to the handles themselves is
the domain of the sysop.

file    bind{} to IO, defaults to 0
dstfile bind{} to IO, defaults to 0

: file$ ( -- )
  Reset "file" and "dstfile", closing them if necessary.

: >file ( hdl -- )
  Close current "file" and rebind it to "hdl".

: >dstfile ( hdl -- )
  Close current "dstfile" and rebind it to "hdl".

: file>dstfile ( -- )
  Close current "dstfile", set it to "file" and reset "file".

## Path literals

There are 2 words for path/file shortcuts: p" and f". These literals work both
in runtime and compile time mode. In compile time mode, it's not the FS ID that
is compiled, but the path string. In other words, path finding isn't done at
compile time but at runtime.

p" foo" is a shortcut for:

  " foo" bootfs :find#

f" foo" is a shortcut for:

  p" foo" bootfs :open >file

## Drive API

The Drive struct offers a unified API to read and write to mass storage.  It
does so in blocks of bytes of a definite size which we call "sector".  Sectors
in the Drive API are organized in a linear manner (LBA, Logical Block
Addressing). If the underlying mass storage driver interfaces with a CHS
(cylinder, head, sector) system, it's the driver's responsibility to convert LBA
to CHS.

Fields:

secsz   size in bytes of each sector read and written by the driver.
        It's often 512.
seccnt  Number of sectors in the drive

Methods:

:sec@ ( sec dst drv -- )
  Read sector "sec" in memory address "dst", which needs to be a buffer at least
  "secsz" in size. "drv" is a pointer to a Drive structure.

:sec! ( sec src drv -- )
  Write sector "sec" from memory address "src".

## File API

The File struct extends IO [doc/sys/io] and represents an opened file. You can
open a file through "Filesystem :open".

A File is a seekable IO that has a size.

To allow file cursors to be reused, the ":close" method needs to be called.

Fields:

used?  Whether this struct slot is currently used. If not, this area of memory
       can be used to open a new file.
fs     Filesystem struct that spawned this file.
pos    Current position of the file, in byte offset relative to the file's
       beginning.
size   Size in bytes of the file. Is updated when writing to the end of the file
       or when resizing.
secwin Sector buffer for this file. This is a SectorWindow struct and has a
       ":secwin" bind{}.

Methods:

:place ( self -- ?a n-or-0 )
  Place the sector buffer so that it points to the current value of "pos". If
  "pos" doesn't have a corresponding sector on disk, n=0 and "a" is absent.
  Otherwise, read the appropriate sector into the buffer and yield the address
  representing "pos" as "a". "n" is nonzero and represents the number of bytes
  following that address that is a valid representation of the corresponding
  sector.

  For example, if the file's first data sector is 42, with a sector size of 512
  bytes and a "pos" of 123, then the 512 bytes of sector 42 are loaded into
  memory, let's say at address $1234. The result of :place will then be
  a=$1234+123 and n=512-123.

  The address thus yielded is expected to have a lifetime tied to the File
  struct. This means that it should be valid until the next :place call on that
  particular file.

:dirty! ( self -- )
  Set current buffer as dirty, that is, in need of sync with mass storage.

:resize ( n hdl -- )
  Change the size of target I/O structure, in bytes.

:flush ( hdl -- )
  If the hdl's buffer is "dirty" (has been changed compared to the permanent
  storage it represents), save that data to permanent storage and flag the
  handle as clean.

Words:

:hold ( self -- )
  Set "used?" to 1.

:release ( self -- )
  Set "used?" to 0.

:seek ( n hdl -- )
  Change current position to "n", which is a byte offset.

:truncate ( hdl -- )
  Truncate the I/O to current position, inclusively. That is, a read operation
  directly following a :truncate will always be EOF.

:maxn ( hdl -- n )
  Yields the number of bytes between "pos" and the end of the file.

:new ( fs -- file )
  Create a new File struct tied to Filesystem "fs".

## Filesystem API

The Filesystem struct encapsulate the logic for interacting with a particular
file system. It has many methods that need to be implemented by the extending
struct and supplies a sizeable set of convenience words on top of those methods.

Moustaches:

drv        Drive pointer
flags      b0=writable
lastcursor Linked list of File structs tied to this filesystem.
{:}curdir  FS ID of current directory
finddir    FD ID of the parent of the last :find result

Methods:

:child ( dirid name fs -- id )
  Enumerate the contents of directory "dirid" and look for name "name". If
  found, returns the ID of the child (either a file or directory), otherwise,
  return 0. See below for the concept of "ID".

  If "name" is "..", yield the parent of "dirid" instead.

:children ( id fs -- ... n )
  Given a parent id "id", yield an unordered list of the children it contains.
  "n" is the number of children and it's followed by that many number of FS IDs
  representing the children of the parent "id". The "." and ".." elements of a
  parent element are never present in that list.

:info ( id fs -- dir? size name )
  Yields "id"'s name, size in bytes and whether it's a directory. The "name"
  part is a string that might be in a static buffer and is only valid until the
  next ":info" call.

:open ( id fs -- file )
  Open file at path and return a handle (a "File" structure) through which other
  file-related word identify the target file. Once a file isn't used anymore, it
  should be closed with :close. Aborts on error.

:newfile ( dirid name fs -- id )
  Create a new empty file named "name" in "dirid" and return the ID of the new
  file. Aborts on error.

:newdir ( dirid name fs -- id )
  Create a new directory named "name" in "dirid" and return the ID of the new
  directory. Aborts on error.

:remove ( id fs -- )
  Remove the element corresponding to "id" from the filesystem.

:newcursor ( fs -- file )
  Allocates a new File struct and yield it.

Words:

:writeable? ( fs -- f )
  Is filesystem flagged as writeable?

:findfreecursor ( self -- file )
  Search in the "lastcursor" LL for an unused File struct. If it finds one,
  flag it as used and yield it. Otherwise, allocate a new File struct, add it
  to the LL, flag it as used and yield it.

:find ( path self -- id-or-0 )
  Recursively look for string "path" in directory "curdir" with the lookup logic
  decribed above in "Path finding logic". If "path" starts with a "/", we begin
  the lookup at the root of the filesystem.

:exec< ( w id self -- )
  Wraps :open and IO :exec< in the same word, with the addition of a nice side
  effect: while it executes "w", it changes "curdir" to the current "finddir".
  "curdir" is restored after the execution ends.

:?newfile ( dirid name fs -- id )
  Create a new file if it doesn't exist. If it exists, return its ID.

:?newdir ( dirid name fs -- id )
  Create a new directory if it doesn't exist. If it exists, return its ID.

:copyfile ( dstid dstfs srcid srcfs -- )
  Copy file "srcid" residing in FS "srcfs" to existing file "dstid" residing in
  FS "dstfs".

:copydir ( dstid dstfs srcid srcfs -- )
  Recursively copy directory "srcid" residing in FS "srcfs" to existing
  directory "dstid" residing in FS "dstfs".

:chdir ( dirid fs -- )
  Change "curdir" to "dirid".

:new ( drv -- fs )
  Allocate a new Filesystem struct targeting drive "drv".

## Global words

The file subsystem also supplies a set of global convenience words not tied to
any struct.

f<< ( "path" -- )
  Find path then interpret is as Forth source, then mark is as "loaded".

?f<< ( "path" -- )
  Find path then check if it's already marked as "loaded". If not, load it and
  mark it.

require ( "path" -- )
  Find path then check if it's already marked as "loaded". If not, abort with
  an error message.

exec<< ( w "path" -- )
  Find path then hijack the input stream so that it yields its contents, then
  call word "w". After "w" returns, close the opened file.
