This file documents the extensions provided by ANICI version 2.0.3.

Anici (pronounced `anarchy') is an extension of Tim Long's ICI by Andy Newman
(andy@research.canon.com.au). Anici adds a host of features to ICI and these
should be considered experimental and not part of ICI proper. ANICI provides,

	o	object serialisation
	o	dynamic loading of native code extensions
	o	debugging interface
	o	user-defined "types"
	o	more math functions
	o	a few new miscellaneous functions and operators
	o	search paths for files referenced via include() and use()
	o	use of assign() and del() for set insert and delete
	o	regular expressions may be added and indexed to return
		the original pattern

ANICI adds the following intrinsic functions,

	save		Save an object to a file.
	restore		Restore an object from a file.

	dload		Dynamically load a native code library.

	typedef		Create a user-defined "type".
	new		Create an instance of a user-defined "type".

	debug		Enable or disable debugging callbacks.

	autos		Return the current auto scope.
	statics		Return the current static scope.
	externs		Return the current extern scope.

	use		Import an ICI module.

	toupper		Return an upper case version of a string.
	tolower		Return a lower case version of a string.

	iciinfo		Return information on the ICI environment.

The `new' and `typedef' functions are defined in a dynamically loaded
module in the `usertypes' directory and are not part of anici proper.
See the section on user-defined types below for details.

In addition to these new functions there are two pre-defined
variables, true and false, and a new operator,

	regexp + regexp -> regexp

Adding two regular expressions produces a regular expression that
is the concatenation of the two operands. This is performed by having
The interpreter remember the pattern string that was used to produce
a regular expression and concatenating the strings and compiling a
new expression when they are added. You can also access the strings
by indexing a regexp by the key "pattern". This returns the string
used to compile the regular expression. This works for both literal
and non-literal forms (i.e., compiled by calls to regexp()) of regular
expressions.

The new math functions were added to allow a Tk/Tcl native code library
to link on SunOS with its static math library causing much headaches.
Anyway the new math functions are hypot, sinh, cosh and tanh. They work
exactly as in C.

The assign() and del() functions may also be used to insert and remove
elements from sets. This was added to make code more readable. Assignments
of one and zero to set elements still acheives the same result however
are far harder to read. The assign() and del() versions are slower
however as they involve a function call operation.

The include() intrinsic has been modified to search $ICI_MODULE_PATH
for the file being included. Unlike use() include() does not add any
extension to the filename argument.

Anici is, apart from these extensions, a standard ICI implementation
i.e., it is an extension to the version of ICI available via FTP
from ftp.research.canon.com.au in the directory pub/misc/ici.

Note that only the BSD, SunOS, Solaris, Linux and Win32 configurations have
been changed as these are the only platforms available to me. I mainly
use FreeBSD for anici development so the BSD port is the most reliable.
Other ports should copy the changes made to the Makefile and conf-$PORT.h
for a port. there are a few new macros to control feature inclusion and a
couple of new source files to be compiled. If you read the patch file it
should be obvious :-). The name of the executable is changed to `anici' but
the name of the system-specific library file is not changed (i.e., you get
libsun.a on SunOS).


Object Serialisation

Anici adds object serialisation - the ability to save an object to
a stream (file) and to later read that object from a stream. The
object serialisation works with almost all ICI object types and only
files, sockets and windows being unable to be archived (these have
external state). Object serialisation uses ICI files as its streams
so objects may be serialised to operating system files, sockets and
other ICI file types that support writing.

Object serialisation has a very simple user interface. Two functions
are added, save and restore, that write and read, objects to and from
files. The functions have the following usage,

	      save(any [, file])
	any = restore([file])

The save function writes a representation of its first argument to
the file object, its second argument. If the file object is not
specified it defaults to the current value of "stdout". The output
stream should be in binary mode on systems where this matters (the
default use of stdout is a little troublesome on such systems).

The restore function reads from a file, which defaults to the current
"stdin", and returns the object found. It complements save and also
requires the stream to be in binary mode. If end-of-file is encountered
when restoring an error ("end of file in restore") is raised. This
allows the NULL object to be restored.

The protocol used to save objects is quite straightforward (although
I don't have time to write about it here) and sends a single byte `tag'
to identify the type of object followed by a type-specific representation.
Objects that may contain references to themselves, either directly or
indirectly, have a "name" associated with them and later references to
the same object within the scope of the save operation refer to the
previously transmitted object allowing cyclic data structures to be
safely saved and restored. An object's "name" is simply its address
within the memory space of the interpreter issuing save. To ensure
that all object types implement serialisation the basic ICI type
structure, the type_t data structure, is extended to include save and
restore functions and any types added to the base interpreter will
need to be modified to include such functions. It should be noted that
when an ICI function is saved the internal compiled form of the function
is written to the stream but when an intrinsic function is saved only
its name is saved. When restoring an intrinsic function the name is
looked up in the current scope and whatever object results is used.
This allows an interpreter to provide alternate implementations of
intrinsic functions when running functions restored from untrusted
sources.

The protocol uses a binary representation with a well-defined byte order
(big-endian) for multi-byte items. This was done for speed of implementation
and it is planned to be altered to use XML in a future version to avoid
problems due to differences in machine architectures (i.e., floating point
formats) and to leverage the large number of XML tools that are expected
to be developed.


Dynamic Loading of Native Code Extensions

Anici adds the `dload' function which supports the dynamic linking of
native-code extensions at run-time. Dload takes a string, the name of
the library to be loaded and loads the library using the native
operating system's dynamic linking facility. Once the library is
loaded dload calls a well-known initialisation entry point
(<library-name>_library_init) and returns the object returned by that
function. The initialisation function can return any type of object
but the convention is to return a struct that contains the objects
exported by the library. The struct is keyed by strings, the names of
the objects exported by the library. This format provides the highest
ease of use as it allows normal "dot notation" to be used to access
objects in the library. An example makes it a little clearer,

	/*
	 * Load the libTest.so code library.
	 */
	extern Test = dload("Test");

	/*
	 * Call a function in the loaded library
	 */
	printf("%s\n", Test.func1());


A simple addition to existing native-code modules makes it possible
to modify existing code to be dynamically loaded and used in the
above fashion without modification to the existing code in the module
(i.e, you just add the well-known initialisation function). No provision
has been made for calling routines that do not have ICI "wrapper" functions
although it is relatively straight forward to add such capability
(i.e., something like Perl's pack/unpack functions and a new intrinsic
for calling C functions with complex arguments specified in an ICI mem
object where `complex' means something other than a string, integer or
float - e.g., an array of char, handling of structures is another matter
and a general solution to the C struct to ICI representation problem is
complex).

ANICI uses a search path to locate shared objects to be loaded. The
default search path is set when ANICI is compiled and typically only
looks in the "/usr/local/lib/ici" directory. The environment variable
ANICI_DLOAD_PATH may be set to a colon-delimited (semi-colon delimited
on Win32 platforms) list of directories.


User-defined Types

User-defined "types" are supported via a dynamically loaded native
code module. The module serves as an example of dynamic loading
while at the same time providing useful functionality. The directory
"usertypes" within the anici source directory contains the module.
a simple test/demonstration program and a Makefile for SunOS.

What I am calling "user-defined types" aren't types in the strict
sense of the word but allow control over the use of ICI struct
objects. A user type allows the programmer to provide functions to
initialise a new instance of a struct and to control the assignment to
and fetching of values from within the struct (i.e., the ICI indexing
operations).

The user creates a new "type" by a call to the `typedef' function which
constructs an object representing the user-defined type. This object
is known as a `utype'. When creating a utype the user specifies the
name of the new type and optionally supplies (ICI) functions that perform
the construction of new objects, the assignment of values to members within
the new type and the fetching of values from the instances of the type.
If any of the functions are omitted the user type behaves as an
ordinary ICI structure (apart from typeof() returning its name rather
than "struct").

The `new' function takes a utype and constructs a new instance of that
type. If the utype defines a construction function that function is
called to initialise the type. When a member of the new type ("controlled
structure" is possibly a better name for these objects) is assigned
to and the type has an "assign" function then this function is called
with three parameters, the object, the key and the proposed value.
The assign function has total control over how values are assigned to
elements of type and can validate the keys used, ensure correct value types,
etc... In a similar manner when the user's type is indexed the type's
"fetch" function is called with the object and the requested key object.
The type's fetch function can validate the key or take whatever action is
desired to return some value. For instance the fetch function could
synthesize a value and not actually store the value or perform I/O to
obtain the value.

As well as providing the dynamic link library (and source) the `usertypes'
directory contains a simple example of a dummy complex number type. The
file Complex.ici defines a simple type that could be used to represent
complex numbers and demonstrates the use of a user defined type.


Debugger

Anici adds a debugger for ICI. The debugger is implemented as a small
extension to the interpreter which adds support for a debug function
interface. With debugging enabled the interpreter will call back to
user defined functions when certain "debugging events" occur. Debug
events are things like executing a statement, calling a function, etc...
The interpreter calls user-supplied ICI code to handle the event and
if the user doesn't supply a function that function is ignored. This
allows the debug interface to be used for more than just debugging,
e.g., statement counting or function tracing. The debug interface can also
be used for program specific debugging as it simply calls ICI functions.
These functions can be an integral part of a program with code to look
for specific actions (such as modifications to a variable).

The interpreter has a new intrinsic function, `debug', that is used to
get and set the current debug "status". The debug status is an integer
value which is non-zero if debugging is currently enabled and zero if
disabled. When called with no arguments debug just returns the current
debug status, when called with a single integer argument debug sets the
debug status and returns its previous value. When debugging is enabled
the interpreter calls the following functions if they are defined,

	_debug_src		Called when each new line is being
				executed. The arguments are the filename
				and line number and the current scope
				of the code executing the statement.
	_debug_call		Called when a function is being called.
				The arguments are the function object
				being called, an array containing the
				actual parameters to the function, an
				integer flag which is non-zero if the
				function is an intrinsic function and
				a struct which is the scope of the
				code calling the function.
	_debug_return		Called when a function returns. The
				arguments being the functions returned
				result (NULL for procedures or functions
				that do not return a value) and the
				scope into which it is returning.
	_debug_error		Called when the program being debugged
				raises an error. The arguments are the
				string error message and the scope.
	_debug_watch		Called when a variable is assigned to.
				The arguments are the "container" object
				holding the key to be assigned to, the
				key, the value and the current scope.


A supplied file, debug.ici, is a first attempt at a command line debugger
using the debug function interface and is intended to provide the most
commonly used functionality of debuggers like gdb and dbx. It is far from
complete, has serious bugs and will change. If you'd like to fix by all
means do so and please share your results.

Using debug.ici to debug ICI programs is simple. You just add code to
your programs to include the file (or `use' it). If the file is installed
in the ICI "library" directory it can be included with a "-ldebug" command
line option to the interpreter. When the program runs a prompt is written
to stderr and commands may entered (from stdin which will break many
things and should be fixed - this is one of those serious bugs I mentioned).
The "help" command is a good starting point.

To make the job of writing debuggers easier there is also a new parsing
function, dparse(), that enables debugging for the duration of a parse
operation. Otherwise it is identical to parse(). Dparse() is mainly of
use of debugging programs that do not wish to have to handle their own
execution within the debugging callback functions.

Miscellaneous Additions

A few miscellaneous functions have been added. These are of main use to
those writing some intricate little bits of ICI and replace slow and/or
complex functions. The functions are,

	autos
	statics
	externs

		These all return the struct object for the corresponding
		scope level. They are provided to avoid pieces of code such 
		as, super(super(scope())) and make things more readable in
		pieces of "system" code. The "autos" function is superfluous
		as it duplicates the existing scope() function.

	use

		This is used to parse a module into its own scope. It
		is similar to the `include' intrinsic but parses the
		module into a new scope so the module has its own static
		scope. The extern scope for the module is the static scope
		of the caller (this is what you really want most of the time).

		The use() function looks down a search path for the module
		to be loaded.

	toupper
	tolower

		Convert a string to either an all uppercase (toupper)
		or lowercase (tolower) format. These are provided for
		performance reasons as doing lots of conversions using
		explode, a loop and implode can be quite slow.

	putenv

		Used to add entries to the program environment. Can be
		called with one or two parameters as in,

			putenv("PATH=/usr/bin:/bin")

		And,

			putenv("PATH", "/usr/bin:/bin");

I've also added some pre-defined variables,

	true
	false

		These are pre-defined variables that refer to the integers
		one and zero respectively. They aren't really needed but
		sometimes provide easier to read programs. As in C be
		very careful about testing for equality to true and note
		that ICI has two representations for false truth values
		(both NULL and 0 are regarded as false).


Operator Changes

    Indexing arrays by arrays.

	I've added the ability to index an array by an array which
	returns an array of the elements from the first array with
	the indicies contained in the second. The array used as the
	index must only contain integers. Inspired by George Politis
	(and APL).

    Adding regular expressions.

	In ANICI it is legal to do <regexp> + <regexp> the result
	being the regexp that results from the concatenation of the
	two regular expressions (which is not checked for legality).

    Indexing regular expressions.

	In addition to the above it is possible to index a regexp
	object with the string "pattern" to return the string that was
	the source of the regular expression (which is how regexp
	addition works in practice).

    del() and assign() may be used with sets.

	In addition their use with struct objects I've made del()
	and assign() work for sets too. Assign() may be used to add
	an object to a set and del() used to remove objects from sets.
	This makes such code more readable than using assignment of
	zero and one to perform these actions.

    The sort() function has a default comparison routine and now
    returns its array parameter to simplify use.
	
	The sort function has been made a little easier to use by
	giving it a default comparison function that simply uses
	the less than and greater than operators on its parameters
	to determine their relative order. Also sort now returns
	the array being sorted making for more natural use in some
	situations. Old programs are unaffected by these changes.
	To see the benifit of the change compare the two pieces
	of code to return a sorted directory listing,

	    Old

		d = dir();
		sort
		(
		    d,
		    [func (a, b) {
			return a < b ? -1 : (a > b : 1 : 0);
		    }]
		);


	    New

		d = sort(dir());

	Which do you prefer?

    Function names.

	In ANICI any function object may be indexed with the string
	"name" to return a string that is the function's name or the
	empty string if the function has no name.

Ownership

The ANICI modifications are NOT in the public domain. I'm keeping the
copyright on this code but allowing anyone to use it for whatever means
they wish, including commercial usage. I do request that users give me
credit though and monetary considerations would be appreciated. So,

	The Anici modifications are copyright (c) Andy Newman, 1996-1998.


Andy Newman, 10 November, 1996.
