$NetBSD: testframe.txt,v 1.9 2023/12/03 09:42:36 rillig Exp $

CURSES TESTFRAME
----------------

1. Introduction

The curses library is a complex piece of software and, often, changes
made to the library may introduce subtle bugs that are hidden by other
actions so a visual check of the curses output may look correct in
some circumstances and the bug only shows itself after a certain
sequence of actions.  To assist with validating that changes made to
the curses library have no undesired effects, an automated test is
needed to detect and highlight any changes in the curses application
output stream.  The programmer can then analyse the output changes and
either correct a bug or update the automated test to accept the new
output as valid.

2. Architecture

The curses testframe consists of two separate programs connected by a
number of pipes and a pseudo-tty (pty).  The programs are called the
director and the slave.  The director reads a configuration file of
tests to perform, passes these commands to the slave over a pipe and
reads the pty for any output from the slave.  Data from the slave is
compared against expected output held in a file and any differences
are highlighted to the tester.  The slave is a curses application that
is forked by the director on start up.  It reads commands from the
director over a pipe, these commands are calls to curses routines
along with the parameters required for the call.  The slave takes the
parameters and uses them as arguments for the requested curses routine
call.  The return value from the curses routine is passed back to the
director over another pipe, if the curses routine updates any passed
by reference arguments then these are also passed back to the director
for analysis.

3. Director

The director has the following optional command line options:

    -v	     	     enables verbose output to assist debugging
    -s slave_path    the director will execute slave_path as the slave
       		     process.  The default is ./slave
    -t term	     Sets the TERM environment variable to term when
       		     executing the slave.  The default is atf

There is one mandatory command line parameter, that is a file name
that contains the test command file.  The test command file holds the
calls required to exercise a particular curses routine and validate
both the return codes from the routines and the output from the
slave.  The test language has a small number of commands, they are:

assign:
      Assign a value to a variable.  The syntax is:

      	     assign var_name value

      Where var_name is the name of the variable.  Variable names are
      an arbitrary sequence of alphanumeric characters, the variable
      name must start with an alphabetic character. Value is the value
      to be assigned.  The value can either be a numeric or a string
      type.  Variables are created on first use and will be
      overwritten on each subsequent use.

call, call2, call3, call4:

      All these are used to call curses routines.  The only difference
      between them is the number of expected return values.  Call
      expects one return value, call2 expects 2, call3 expects 3 and
      call4 expects 4.  Any parameters that are passed by reference and
      updated by the call are treated like returns and are listed after
      the regular return value.  The general form of a call is:

		call[234] <return...> <function> <argument...>

      As an example, the macro 'getyx' has 3 parameters, as seen by a C
      programmer, and it has return type 'void'.  If its return type
      were not void, that would be its first return value.  The first
      parameter is a window and is passed by value.  The other two, 'y'
      and 'x' are references to return values, thus they do not count as
      parameters.  In summary, there are 2 return values, therefore
      'call2' is used:

		call2	4 5	getyx		$win1

      The two return values from the actual function call are validated
      to be '4' and '5'.  The parameter '$win' refers to a variable that
      has been defined by a previous call or variable assignment.

      Instead of directly validating the return values, they can also be
      assigned to variables:

		call2	y x	getyx		$win1
		check	y 4
		check	x 5

      Another example is 'mvwaddwstr', which has a single return value
      (OK or ERR) and 4 parameters.  It is called like this:

		call	OK	mvwaddwstr	$win1 4 5 "text"

      Variables are referenced in a call parameter list by prefixing the
      name with a '$' sign.  All returns are validated against the
      expected values and an error raised if there is a mismatch.  The
      only exception to this is when the return is assigned to a
      variable.  Valid values for the returns list are:

      	  	  variable - assign the return to the given variable
		             name.
      	  	  numeric  - the value of the return must match the
      	  	  	     number given.
		  string   - an arbitrary sequence of characters
      	  	  	     enclosed in double quotes.
		  ERR      - expect an ERR return
		  OK	   - expect an OK return
		  NULL	   - expect a NULL pointer return
		  NON_NULL - expect a pointer that is not NULL valued

      There is one special parameter that can be passed to a call,
      that is the label STDSCR.  This parameter will be substituted by
      the value of stdscr when the function call is made.

cchar:
      Defines a variable as a complex character type (cchar_t).  The
      syntax is:

                cchar var_name attributes elements

      Where attributes is the attributes for the complex character, this
      can be an integer, a variable or bitwise or'ed list (see below
      for a description of the syntax). The elements is one or more wide
      characters that make up the elements of the complex character.  A
      single element of either a numeric, variable, byte or single
      character string can simply be wrtten at the end of the line.
      Multiple elements need to be space separated and enclosed in
      square ([]) brackets, the types allowed are the same as for the
      single element case.  In addition an element can be repeated by
      following an element with a '*' then a number which will cause
      the last element to be repeatedly added to the array the number
      of times specified by number.  Once the complex character variable
      has been defined it can be used in calls to curses routines
      requiring a complex character type argument.

check:
      Validate the value of a variable.  This allows a variable to be
      checked for an expected return after it has been assigned in a
      previous call.  The syntax is:

      	       check var_name expected_result

      Where var_name is a variable previously assigned and
      expected_result is one of the valid return values listed in the
      above call section.

compare:
      Compares the output stream from the slave against the contents
      of a file that contains the expected
      output.  The syntax is:

      	       compare filename

      Where filename is the name of the file containing the expected
      output.  The file can either be an absolute path or relative
      path.  In the latter case the value of the environment variable
      CHECK_PATH will be prepended to the argument to provide the path
      to the file.  The contents of this file will be compared byte by
      byte against the output from the slave, any differences in the
      output will be flagged.  If the director is not in verbose mode
      then the first mismatch in the byte stream will cause the
      director to exit.

comparend:
      Performs the same function as the above compare except that
      excess output from the slave is not discarded if there is more
      data from the slave than there is in the check file.  This
      allows chaining of multiple check files.

delay:
      Defines an inter-character delay to be inserted between
      characters being fed into the input of the slave.  The syntax
      is:

		delay time

      Where time is the amount of time to delay in milliseconds.

include:
      Include the contents of another test file, the parser will
      suspend reading the current file and read commands from the
      include file until the end of file of the include file is
      reached at which point it will continue reading the original
      file.  Include files may be nested.  The syntax is:

      	     	include filename

      Where filename is the name of the file to include.  If the
      filename is not an absolute path then the contents of the
      environment variable INCLUDE_PATH are prepended to the file
      name.

input:
      Defines a string of characters that will be fed to the slave
      when a call requires input.  Any unused input will be discarded
      after the call that required the input is called.  The syntax
      is:

		input "string to pass"

noinput:
      Normally the director will error if an input function is called
      without input being previously defined, this is to prevent input
      functions causing the test to hang waiting for input that never
      comes.  If it is known that there is pending input for the slave
      then the noinput keyword can be used to flag that the input
      function has data available for it to read.  The noinput command
      only applies to the next function call then behaviour reverts to
      the default.

The testframe can define different types of strings, the type of string
depends on the type of enclosing quotes.  A null terminated string is
indicated by enclosing double (") quotes.  A byte string, one that is
not null terminated and may contain the nul character within it is
indicated by enclosing single (') quotes.  A string of chtype
character which are a combined attribute and character value is
indicated by enclosing backticks (`), for this type of string pairs of
bytes between the backticks are converted to an array of chtype, the
first byte is the attribute and the second is the character.

All strings defined will have a simple set of character substitutions
performed on them when they are parsed.  This allows the tester to
embed some control characters into the string.  Valid substitutions
are:

	\b	backspace
	\e	escape
	\n	new line
	\r	carriage return
	\t	tab
	\\	\ character
	\nnn	Where nnn is three octal digits, the character
		represented by the octal number will be inserted into
		the string.

Integers may be specified by either a plain numeric (e.g. 12345) or by
hexadecimal notation by prefixing the number with 0x (e.g. 0x3039).
Internally, no distinction is made between the two formats and they
can be freely intermixed.

Integers and variables containing integers can have operations
performed on them.  Currently only bitwise ORing numbers together is
supported.  This can be done by separating a list of integers and
variables with the pipe (|) symbol and enclosing the entire list in
round brackets "()" like this:

      ( $var1 | 0x0100 | $var2 | 512 )

Variables and integer constants may be freely intermixed.  The result
of the operation can either be used as an argument for a call or can
be used as an expected result for a call.

In addition to all the curses calls being supported by the slave,
there is one more special call called "DRAIN".  This call repeatedly
calls wgetch(win) until there are no more characters in win.  The call
assumes that the curses input is either in no delay or timed input
mode otherwise the test will time out and fail.  This call can be used
to clear any pending input when testing a timed read, to prevent the
input from being used in a later test.

4. Slave

The user has no direct interaction with the slave process.  The slave
is forked off by the director and communicates to the director over a
set of pipes and a pseudo-tty connected to its standard i/o file
descriptors.  The slave executes the passed curses calls and passes
back return values to the director.  The slave automatically calls
initscr() on startup.
