

Marc Mertens

LISP DEBUG

Introduction

LISP DEBUG is a source level debugger, stepper and profiler for LISP
programs. The current implementation works with GCL 2.2 or higher,
CMUCL Lisp (version 24-04-99 or higher), ACL5 and CLISP (version 22-07-99
or higher) and is tested on LINUX 2.0 (the current GCL version of
LISP DEBUGGER doesn't work on SUN Solaris). Although LISP DEBUG makes
use of the TCL/TK system for the GUI, no TCL/TK extensions are needed
for the LISP system (except for GCL version). LISP DEBUG has the following
features :

* Source level debugger (highlight in a sourcewindow the code being
  executed).

* Lispform oriented not line oriented (function call is highlighted
  not the line).

* Multisource debugging is possible (the system will switch to another
  source if needed).

* Controls :

  * Step (execute the current form and stop before the next form ).

  * Step over (execute the current form and stop before the next form
    in the current list).

  * Next (execute the current form and stop at the highlighted form).

  * Continue (execute the current form and don't stop anymore)

* Breakpoints and conditional breakpoints are placed on lisp forms
  (not on lines).

* Placement of watchpoints on variables and lispforms is possible.

* Evaluation of lispexpressions is possible during debugging.

* Timetraveling (the debugger keeps track of the lisp forms executed
  + their environment , this makes it possible to step backwards and
  forwards through time).

* Profiling (count how many times a lisp form is executed).

* Halting is possible before and after executing of a form.

* If halted after execution of a form then:

  * The result of the execution of the form is displayed

  * It is possible to change the value of the execution.

* Extensible (you can extend the lisp control structures the debugger
  recognizes by use of syntax definitions).

* If programs loaded in the debugger encounter during execution a continuable
  error, the lisp debugger is entered, the call causing the error
  is highlighted and you can give the correct result of the call and
  continue executing the program (this feature doesn't work in gcl
  because gcl lacks handler-case , needed for this feature).

LISP DEBUG works by instrumentation. When a lisp source is loaded in
the debugger the following is happening :

1. A tempory file is created which contains the lisp source with debug
  code added to it (essentially code to call the debugger, to save
  position info and to save the environment of the function call).

2. The tempory file is loaded in the LISP system (you can ask to compile
  it before the load).

3. The source code is loaded in a sourcewindow to be displayed during
  the debug process.

4. You can now use the sourcewindow to set breakpoints ,watchpoints
  ... . When you run now code in the LISP system the execution will
  stop at the breakpoints and the lispform which should be executed
  next is highlighted in the sourcewindow. From then on, you can fully
  control the execution of the code. 

5. If during execution of the debugged code a continuable lisp error
  happens, the debugger is entered and the call causing the error
  is highlighted. If needed you can give the result of the call to
  the faulty code and continue running your program. 

6. When you close a source the source will be removed from the sourcewindow
  and the original code (without instrumentation code) is loaded in
  the LISP system.

The original source code is never touched.

1.1 Warnings

* Adding instrumentation code has some risks associated with it.

  * Bugs in the debugger can cause malfunction of your code (I have
    done my best to avoid bugs but nobody is perfect :().

  * Bugs in the debugger can make that not every part of your program
    is available for debugging.

  * Bugs in the compiler/optimizer of lisp can make that there is a
    different behaviour of debugged programs and non debugged programs.

  * Debugged code is much slower then non debugged code.

* If you work with recursive data which is impossible to print in LISP,
  the debugger will also go in a infinite loop to try to print this
  data. Turning off recursive printing in LISP will also solve the
  problem for the debugger (as he uses the print functions of LISP)
  as long as the results are not displayed in a separate window.

* Macros are not easy to debug , given a macrocall the system will
  try to expand it and then try to add debugging code to the expanded
  code. System macro's which represents a special controlstructure
  (like cond,if ...) are handled in a special way. If you define your
  own controlstructures you can extend the debugger to handle them
  in the correct way.

* Breakpoints can be placed on every lispform you can evaluate (with
  the exception of atomic expressions and some macros (example (defgeneric
  ...), (function ...)).

* If you can't place a breakpoint on a form it is usually possible
  to place a breakpoint on certain subforms.

* If you make changes to the source , you have to reload the source
  in the debugger (you can use copy/paste to load sources in the debugger
  if you use emacs or xemacs).

* If there are bugs in your source code which causes a halt during
  loading , the debugger will halt when he loads the modified source
  code and you will enter the native debugger. If you exit the native
  debugger the rest of your code will be loaded. Always make sure
  that you can load code before you try to debug the code.

* The debugger sits in package ``DEBUGGER'' , don't use this package.

* In the ``USER'' package the function name ``debug'' (``deb'' in ACL)
  is exported , so you may not use this name for a function.

* Allegro uses the name ``debug'' so I used ``deb'' in the case of
  Allegro.

* If you choose to compile the debugged code in CMUCL it can take a
  long time and lots of memory because the addition of the extra debugging
  code.

* If you choose to compile the debugged code in ACL you can get errors
  because the not correct (in my opinion) handling of ACL of macros
  used in optional parameters of functions.

Overview of the debugger.

2.1 Quick procedure to get you running.

1. Make sure that X Windows is running,

2. Start LISP (standalone or as a process in emacs (xemacs).

3. Type (debug) at the lisp prompt ((deb) for Allegro). A separate debug
  window should open (see Figure: Starting Debugger) . You control
  debugging using this window.([fig] Start Debugger\includegraphics{pic1.ps} )  

4. Load a lisp source in the debugger using <File><Open> .([fig] File Selection\includegraphics{pic2.ps} ) 

5. After loading the source, you can set a breakpoint by first selecting
  a lisp form and then clicking on <Breakpoint>. If a breakpoint is
  possible the breakpointed code will have another foreground color
  (default red). 

  Using the middle mousebutton you select a whole function , double
  clicking select a list.([fig] Source Window\includegraphics{pic3.ps} ) 

6. Return then to the lisp prompt in the LISP system and type an expression.
  If the execution of the expression causes the breakpointed code
  to run , the currently executed code will be highlighted in the
  source window and the debugger waits for your commands. You can
  now step through the code , set watchpoints , evaluate expressions
  ... .([fig] Executing Code\includegraphics{pic4.ps} ) 

7. For more detailed information of using the debugger go to the user
  manual.

8. To stop debugging , use <File><Exit> , the original source (without
  debugging code) will be loaded back in the LISP system. 

2.2 Using the debugger

2.2.1 Starting the debugger.

To start the debugger , first make sure that X Windows is running ,
then start lisp (in a shell , in an emacs (xemacs) process or whatever
editor you want). At the lisp prompt type (debug) ((deb) if you are
using Allegro). A debug window will then start allowing you to control
the debugging process. For CLISP a special prompt is displayed (DEBUGGER::),
this is because the debugger uses in CLISP its own toploop.

2.2.2 Debugging programs.

The debug window consists of the following parts :

* A menu line ('File','Source','Edit','Options','Tools') to Open/Close
  sources , to switch between loaded sources , to paste code to the
  debuggger, to configure the debugger and to do profiling.

* A button panel containing the most used functions of the debugger
  (Stepping , Breakpointing , Watching , Evaluating and timetraveling).

* A source pane which shows the source code , during debugging the
  debugpoints have a red foreground color , the code to be executed
  is highlighted in light blue and if you do profiling, code called
  the specified amount of times is also highlighted in yellow.

* A watch pane, contains the watched variables and expressions in addition
  to their values (this info can also be displayed in a separate window).
  The watch pane can also display the result of an expression evaluation
  or the result of a call before stepping to the next call.

* Command pane, used to type in an expression that can be used as a
  watchexpression , that can be evaluated ,that can be returned as
  the value of call or that can be used in a conditional breakpoint.

To debug a function you follow in general the following steps :

1. Load the sourcecode of the function in the debugger , this can be
  done in two ways:

  (a) The code is defined in a sourcefile on the harddisc, you use then
    <File><Open> to load the whole sourcefile in the debugger. This
    is the preferred way of working because you can then debug all
    related functions in the sourcefile.

  (b) The code is defined in an editor (like emacs , xemacs ...), select
    then the code, use the <Copy> function of the editor to put the
    code in the copybuffer. and use then <Edit><Paste> on the debugwindow
    to load the code in the sourcewindow. (There is a change that
    this is not working because Copy/Paste is not weldefined in X
    Windows). 

  (c) Once code is loaded in the debugger the debugger will be activated
    in the following cases: 

    i. By a continuable lisp error during executing of the debugged
      code. The call causing the error is then displayed and you can
      try to finish the execution of the code by giving a result for
      the faulty call (this is not working in gcl).

    ii. By executing a piece of code with a breakpoint on it (for conditional
      breakpoints the condition must be fullfilled as well).

    iii. As the result of <Step>,<Step Over> and <Step Next>.

    iv. Because you didn't use <Continue> to stop debugging the previous
      lisp expression.

2. Set a breakpoint , this can be done in two ways :

  (a) Select a form in the sourcepane and press then 'Breakpoint' to
    set a breakpoint. The color of the text in the form changes then
    to red to indicate a breakpoint set on this form. The color will
    only change if you can really set a breakpoint . Example : in
    (defgeneric fac (n) (:method ((n integer)) (if (zerop n) 1 (*
    n (fac (1- n))))) (:method ((n string)) (concatenate n ``!'')))
    you can set a breakpoint on (:method ((n integer)) (if (zerop
    n) 1 (* n (fac (1- n))))), (if (zerop n) 1 (* n (fac (1- n))))
    , (zerop n) , (* n (fac (1- n))) , (fac (1- n)) , (1- n) , (:
    method ((n string)) (concatenate n ``!'')) , (concatenate n ``!'')
    but not on the (defgeneric ....) form. It should be clear that
    a breakpoint is not set on a line but on a form like a function
    call ((function arg1 .... argn) , ...) , a special function call
    ((cond (( ....) ...) ...) , (defun ...) , (let ...) , (do ....)
    ....) or a macro call. Breakpoints just like forms can be nested.
    The 'Breakpoint' act as toggle if you select a form which already
    has a breakpoint selected with it you remove the breakpoint. 

  (b) Type in an expression in the command pane , select a form in the
    source pane and press 'Break If' , this sets a conditional breakpoint
    , execution is halted only if the evaluation of the condition
    gives true as a result. An error during the evaluation is considered
    false (a dialogbox explaining the error is displayed in this case).
    The 'Break If' button act as a toggle, just like the 'Breakpoint'
    button.

3. Set a watchpoint , this can be done in two ways :

  (a) Select a variable in a function and then press 'Watch' . In the
    watchpane you will see now the variable followed by an arrow to
    its value. If the variable is not defined the value will be 'Undefined'.

  (b) Select a form in the source pane, type a expression in the command
    pane and press 'Watch exp'. This will set a watchpoint on the
    expression in the command pane, which is evaluated when forms
    in the selected code are executed (otherwise 'Undefined' is used
    as value).

    A common mistake is to forgot to make a specific selection.

4. To unset a watchpoint. Press the results of the watchpoint in the
  watch pane, a dialogbox pops up to ask if you want to display the
  results in a separate window or delete the watchpoint. If you work
  with separate results windows, then deleting these windows causes
  the display of the results in the watch pane.

5. You can go now to the LISP system and type in some lisp expressions.
  If a breakpoint is encountered then the execution of the function
  will be paused and the code to be executed will be highlighted in
  the source pane of the debug window (if needed the corrected source
  will be loaded or a repositioning will take place so that the executing
  code is always visible). You can react now in the following ways :

  (a) Press the 'Step' button and execute the highlighted form, pausing
    at the next form to be executed (this could be a subform of the
    highlighted form). 

  (b) Press the 'Step Over' button, execute the highlighted form, pausing
    at the next form within the parent form (so we will not step through
    code called during execution of the highlighted form (unless breakpoints
    are placed of course)).

  (c) Select the code where the debugger should pause next and press
    then the 'Next' button. The debugger will then start executing
    the code until a breakpoint is encountered or the selected code
    is reached.

  (d) Press the 'Continue' button , the debugger will then start executing
    the code until a breakpoint is encountered or nothing has to be
    executed.

  (e) Press the 'Back' or 'Forward' button to do time traveling (look
    at the previous/next executed code and variable context at that
    point).

    If you have used Step or Step Over and all code is executed, then
    the system stays in a stepped state, if you execute another debugged
    function from the lisp prompted then the system pauses execution
    waiting on you to press 'Step' , 'Step Over' , 'Continue' or 'Eval'.

6. If you have enabled the option 'Display result last call', then the
  debugger stops also just after execution of a call. If stopped after
  execution of a call, the result of this call is displayed and the
  button 'Change Result' is enabled. If you type in a expression in
  the command pane and press 'Change Result' this expression is evaluated
  and returned as result of the call instead of the original result.

7. Select a source via <Source>.

8. You stop debugging of a source by selecting the source and then <File><Close>.
  The original source code is then loaded in the LISP system.

9. Another way of stopping of debugging is the <Tools><Interrupt> menu,
  this puts the executed code in the native debugger and stops debugging
  of the code with the lisp debugger. The original code is not reloaded
  in the LISP system. Use this option with care (it can cause problems
  in certain lisp systems).

10. You stop the debugger via <File><Exit>. Before exiting , the debugger
  will load the original sources of all debugged code back in the
  LISP system.

Reference Manual

3.1 Menus

3.1.1 Overview

Following menu items exist in the debugger :

* <File><Open>

* <File><Close>

* <File><Exit>

* <Source>

* <Edit><Paste>

* <Edit><Find>

* <Options><Color Break>

* <Options><Color After Break>

* <Options><Color Breakpoint>

* <Options><Color Breakpoint If>

* <Options><Color Profiling>

* <Options><Font>

* <Options>[Compile debugged code]

* <Options>[Display result last call]

* <Options><Save Options>

* <Options>[Save on exit]

* <Tools><Start profiling>

* <Tools><Stop profiling>

* <Tools><Interrupt Program>

3.1.2 Detailed description of menu items

3.1.2.1 <File><Open>

Opens a window which allows you to select a LISP source. When pressing
Ok , the selected source is processed to add debugging code and to
save the modified code in a tempory file. If the option <Options>[Compiled
debugged code] is true then the tempory file will be compiled and
the compiled code will be loaded in the lisp system, in the other
case the tempory file will be loaded in the lisp system.([fig] <File><Open>\includegraphics{pic2.ps} ) 

3.1.2.2 <File><Close>

The original code of the file in the sourcewindow is loaded in the
lisp system. The source window is then emptied and filled with the
first loaded source (if it exist).

3.1.2.3 <File><Exit>

Exit the debugger. Before this happens the original version of all
debugged sources is loaded in the lisp system.

3.1.2.4 <Source>

Allows you to select which of the sources loaded in the debugger is
shown in the source window. This menu is detachable.

3.1.2.5 <Edit><Paste>

If in emacs (xemacs) (or maybe your editor of choice) a selected area
is copied to the copy buffer, then this selected area is used to create
a tempory source which is loaded in the debugger. This allows you
to debug functions which are in a editor but not yet in a source file.
(Every copy/paste operation supported by tk_textPaste should work).

3.1.2.6 <Edit><Find>

Activate the search function, to search the source currently displayed
in the source window. Three types of search are possible :

* Case , case sensitive search.

* No Case , case insensitive search.

* Regular expression , allows you to use regular expressions in the
  search.([fig] Find\includegraphics{pic5.ps} ) 

3.1.2.7 <Options><Color Break>,<Options><Color After Break>,<Options><Color
  Breakpoint>,<Options><Color Breakpoint If>,<Options><Color Profiling> 

Allows you to change the color of the current executed code, the code
to be executed next, the breakpoints , the conditional breakpoints
and the profiled code.([fig] Color\includegraphics{pic6.ps} ) 

3.1.2.8 <Options><Font>

Use this menu to change the font used in the source, watch, command
pane and separate result windows.You better avoid a proportional font,
indention is shown much better if you use a fixed font (the default
font is 'fixed').([fig] Font\includegraphics{pic7.ps} ) 

3.1.2.9 <Options>[Compile debugged code]

When this switch is enabled, the source with debugcode added is first
compiled before it is loaded in the lisp system. This could be usefull
on lisp systems which do a precompile of a function before executing
it for the first time (because of the extra debugging code , this
precompile could take a long time). 

3.1.2.10 <Options>[Display result last call]

If enabled the debugger will not only stop just before a call is executed
but also just after a call is executed. Different colors are used
to highlight the executed (to be executed) call. If stopped just after
execution of a call, it is possible to change the result of this call
with the 'Change Result' button.

3.1.2.11 <Options>[Enter debugger in case of error]

If enabled, then during execution of debugged code when there is a
LISP error, the error is displayed in a dialogbox and the debugger
is entered (even if there where no breakpoints). The lisp expression
causing the error is highlighted and you can supply a result for this
lisp expression. 

NOTE: This feature doesn't work for GCL , even if enabled (this is
because handler-case is missing in GCL).

3.1.2.12 <Options><Save Options>

Save the options in <Options> in the file $HOME/.lispdebug.lisp. During
startup of the debugger, this file is always executed. This file is
a standard text file and contains the following settings :

* (DEBUGGER::setting "COMPILE_CODE" 0)

* (DEBUGGER::setting "SAVE_ON_EXIT" 0)

* (DEBUGGER::setting "DISPLAY_RESULT" 0)

* (DEBUGGER::setting "DEBUGPOINT_COLOR" "#ff0000")

* (DEBUGGER::setting "DEBUGPOINTIF_COLOR" "#808000")

* (DEBUGGER::setting "CURRENT_COLOR" "#00ffff")

* (DEBUGGER::setting "AFTER_COLOR" "#008080")

* (DEBUGGER::setting "PROFILE_COLOR" "#ffff00")

* (DEBUGGER::setting "FONT" "-*-helvetica-medium-r-normal-*-120-*-*-*-*-*-*")

The explanation of these codes is as follows :

* DEBUGGER::setting is a LISP function which when called change the
  configuration of the debugger in a way defined by the first string
  (=parameter), the last arg is the value of the parameter.

* COMPILE_CODE if equal to 1, enable <Options>[Compile debugged code],
  otherwise disable this option.

* SAVE_ON_EXIT if equal to 1, enable <Options>[Save on Exit], otherwise
  disable this option.

* DISPLAY_RESULT if equal to 1, enable <Options>[Display result last
  call], otherwise disable this option.

* DEBUGPOINT_COLOR is in rgb values the foreground color of a breakpoint.

* DEBUGPOINTIF_COLOR is in rgb values the background color of a breakpoint.

* CURRENT_COLOR is the color used to highlight a call just before it
  gets executed.

* AFTER_COLOR is the color used to highlight a call just after it gets
  executed.

* PROFILE_COLOR is the color used to highlight code fullfilling the
  profile count.

* FONT is the font used in the debugger.

If you modify or create this file manual (for example because the default
color or font is not supported on your system) be sure to not forget
DEBUGGER (in case of Allegro it must be in uppercase to).

3.1.2.13 <Options>[Save on Exit]

When enabled, the current options are saved in $HOME/.lispdebug.lisp
when the debuggers stops.

3.1.2.14 <Tools><Start Profiling>

Enables the profiling part of the debugger. The profile counters (how
many time a piece of code is executed) are reset to zero and an extra
slider is made visable in the left part of the debugwindow. When code
is executed the system keep track of the number of times a lispform
is executed. By using the slider you highlight the code which is at
least as many times executed as the number shown by the slider.([fig] Debugger when profiling\includegraphics{pic8.ps} ) 

3.1.2.15 <Tools><Stop Profiling>

Disables the profiling part of the debugger.

3.1.2.16 <Tools><Interrupt Program>

Executes an (break ..) command in the lisp system and halts execution
of the code (by gooing in the native debugger). Be carefull with CMUCL
I noticed that to much breaks can break their native debugger, ACL
also gave some problems during interruption.

3.2 Source Pane

3.2.1 Content.

The source pane contains the source of the code which is debugged.
The source which is displayed depends on the following conditions:

* If you open a new source , then this is the source displayed.

* You can change the source displayed via the <Source> menu.

* If you close a source , the first opened source is displayed.

* If the system encounters a breakpoint , the source containing the
  breakpoint is displayed, the window on the source is also repositioned
  so that the executing code is visable.

3.2.2 Selection

Selection is the base for setting breakpoints, setting conditional
breakpoints and setting watch expressions. Selection can be done in
the following ways.

* Dragging. Click with the mouse in the sourcepane (use the left button),
  drag the mouse to an other point and release the button.

* Click somewhere in a function definition with the middle mouse button.
  This selects the whole function.

* Double click with the left button , this selects a whole list.

3.2.3 Highlighting.

Pieces of the sources are highlighted as follows (we use the default
colors of the debugger).

* Red foreground , this is a breakpoint.

* Dark green foreground , this is a conditional breakpoint.

* Light blue background, this code will be executed next.

* Dark blue background, this is just executed code.

* Yellow background , this code fulfills the profile count (is at least
  as many times executed).

3.3 Watch Pane

3.3.1 Content.

The result pane contains the value of watch variables, the results
of evaluating watch expressions, the result of a evaluation, or the
result of previous executed code. The format used is as follows 'exp
--> value' . Here 'exp' could be a variable , a watched expression,
a evaluated expression or ``RESULT' (the result of previous executed
code). 'value' could be 'Undefined' (the variable is not defined or
the expression couldn't be evaluated) , (values .....) the result
was caused by a (values ...) statement or just the value(result) of
the variable (expression). If you press on a 'exp --> value' line,
a dialogbox popups offering you the choice of stopping the watch,
displaying the result in a separate window or just returning.

3.4 Command Pane

3.4.1 Content.

The command pane is the only pane which can be edited. You use it to
type in expressions needed by the debugger. This will be the case
in the following situations:

* Conditional breakpoints (NIL or non NIL of the evaluated expression
  determines if the system stops or not).

* Watch Expression (the expression to watch).

* Eval (the expression to evaluate).

* You want to change the result of a call (with the 'Change Result'
  Button).

3.5 Buttons

3.5.1 Overview.

The following buttons are available:

* <Step>

* <Step Over>

* <Next>

* <Continue>

* <Breakpoint>

* <Break If>

* <Watch>

* <Watch Exp>

* <Change Result>

* <Eval>

* <Back>

* <Forward>

3.5.2 Detailed description of the buttons.

3.5.2.1 <Step>

When a program is halted (before or after execution of the highlighted
call), <Step> allows you to proceed to the next call (or do the highlighted
call) and wait then on user input (before the next call or after the
current call). 

3.5.2.2 <Step Over>

When the program has halted, press <Step Over> to execute the highlighted
code and advance execution. The debugger will halt the program in
the following cases (whichever first occurs).

* A breakpoint is encountered.

* A lispform in the same enclosing list as the highlighted form must
  be executed.

3.5.2.3 <Next>

When the program has halted, use <Next> to advance execution. To use
<Next> first select another lisp form and press then <Next> the execution
with then proceed until one of the following two cases are encountered
(whichever first occurs).

* A breakpoint is encountered.

* The highlighted code must be executed.

3.5.2.4 <Continue>

Pressing <Continue> will advance a halted program , the program will
either fully execute or will halt at the next breakpoint.

3.5.2.5 <Breakpoint>

Use this to set a breakpoint. To set a breakpoint just highlight a
lispform and then press <Breakpoint>. If during executing of code
the highlighted code must be executed then the debugger halts execution
just before this code gets executed. Use <Step>,<Step Over>,<Next>
or <Continue> to continue execution. <Breakpoint> act as a switch,
so if you try to set a breakpoint on a existing breakpoint you are
actually clearing the breakpoint.

3.5.2.6 <Break If>

Use this to set a conditional breakpoint. To set a conditional breakpoint,
highlight a lispform, type a condition in the command window and press
<Break If>. If during execution of code the highlighted code must
be executed then the debugger evaluates the condition and if the result
is non nil execution will halt. <Break If> act as a switch, so if
you try to set a conditional breakpoint on a existing conditional
breakpoint you are actually clearing the breakpoint.

Errors during evaluating the condition counts as a NIL and a message
detailing the error is displayed. Also the condition is only evaluated
in the highlighted code.

3.5.2.7 <Watch>

Sets a watchpoint on a variable. To do this select the variable and
then press <Watch> , the variable and its value is then visible in
the result pane.

3.5.2.8 <Watch Exp>

Sets a watchpoint on a expression. First select a area where the watchpoint
is valid, then type in the expression in the command pane and press
<Watch Exp>. When execution happens in the selected area the exp is
evaluated and the result is displayed in the result pane. If the evaluation
of the expression causes an error this is displayed as an error message
and the result is undefined.

3.5.2.9 <Change Result>

This button is only active if the debugger is entered just after evaluation
of a call (which is highlighted), this can happen because of a lisp
continuable error in the debugged code or because the option [Display
result last call] is enabled. If active you can type in the command
pane a expression and press <Change Result>, the expression will then
be evaluated and the result is used as the result of the displayed
call. Hint: use (values ....) if you want to return more then one
value.

3.5.2.10 <Eval>

Evaluates a expression in the debugging context. If during debugging
(when execution has halted and you have control) you feel the need
to evaluate a expression in the context of the highlighted code do
the following. Type in an expression in the command pane and press
<Eval>, the result will then be visable in the result pane. If the
evaluation of the expression causes an error this is displayed as
an error message and the result is undefined. Warning: evaluated in
the context doesn't mean that the expression can change the context.
(for example you can not change lexical variables but you can use
their values). Be also carefull during time travelling, global values
are not restored during timetravelling.

3.5.2.11 <Back>

Activates time traveling , the debugger keeps track of the 100 lasts
forms executed together with their context. By pressing <Back> you
go to the previous executed form and change the context to the context
at that time. Has the same functionality as the frame concepts in
the system debuggers.

3.5.2.12 <Forward>

Activates time traveling , the debugger keeps track of the 100 lasts
forms executed together with their context. By pressing <Forward>
you go to the next executed form and change the context to the context
at that time. 

Extending the debugger

4.1 Introduction.

LISP is a very extensible language in the way that you can use macros
to define your own control structures. This debugger tries to behave
logical for the control structures as defined in ANSI LISP but doesn't
know how to handle controle structures you define yourself. To solve
this, you can modify the way how this debugger adds instrumentation
code to the source (so that your extensions are covered correctly).
The way I have made this possible is to provide you with a special
language which allows you to express the syntax of lispforms with
a little bit of semantics as well. In fact the core of this debugger
is generated automatically by a source of syntax diagrams defining
the syntax of most special lispforms in ANSI LISP.

If you want to modify the debugger keep in mind the following points :

* Function calls are handled correctly by the debugger, you don't have
  to do anything at all for this.

* Macro's are the problem . The standard lisp macros are covered in
  the debugger and he tries to make the best of the ones you define
  (first he expands the macro call and then tries to add debugging
  code to the expanded code) but the result is not always what you
  expect. It is here that you can modify the behavior of the debugger.

* Modifying the debugger can stops it working, so be carefull. Nothing
  can go wrong if you make sure that you have made backup copies of
  the following files:

  * lispsyntax (this is the source file defining the working of the
    current debugger).

  * debugcode.o,debugcode.x86f , debugcode.fas or debugcode.fasl (this
    is the core of the debugger for GCL,CMUCL,CLISP or ACL5, loaded
    when you start (debug) or (deb)).

4.2 Modifying the debugger

The steps to modify the debugger are as follows:

1. Create a source file defining how the lisp debugger should transform
  sources. You can take the file delivered with the debugger as a
  base and extend it. This file is '/usr/local/lib/lispdebug/lispsyntax'
  (if you have done a default installation of the debugger). Remember,
  this file will redefine the way how the debugger transform sources
  to debugged sources, if you make errors or forget things the debugger
  can act strangely. The syntax and semantics of the language to use
  is described in the next chapter.

2. Use the lisp function DEBUGGER::process-definition-file to compile
  this file to lisp code containing the parse and convertion code
  of the debugger. The syntax to use is either :

  (a) (DEBUGGER::process-definition-file <Source-file>) , the source
    file is compiled to a lispfile (debugcode.lisp) which is then
    compiled by the lisp system and then loaded in the lisp system
    modifying the behavior of the debugger. 

  (b) (DEBUGGER::process-definition-file <Source-file> <Out>) , the source
    file is compiled to a lispfile (with filename <out>.lisp) which
    is then compiled by the lispsytem and then loaded in the lisp
    system modifying the debugger.

3. Once 2 is finished you can test the debugger to see if he handles
  the new definitions well (to test the normal lisp constructs look
  at the testgcl.lisp,testcmucl.lisp,testclisp.lisp and testacl5.lisp
  files). If you are convinced that the debugger is working correctly
  you can make your modifications permanent by copying the object
  file generated in (2) to /usr/local/lib/lispdebug or to the value
  of the LISPDEBUG environment variable. (If you use 2.a. the object
  files are either debugcode.o,debugcode.x86f,debugcode.fas or debugcode.fasl
  for gcl,cmucl,clisp or acl).

4.3 Extension Language Of the Debugger.

The extension language of the debugger is strongly based on the syntax
descriptions used in ANSI LISP, so that writing extensions is a simple
as writing a syntax diagram. Be carefull however, although the language
looks simple their are some know cavecats which should be dealed with.

4.3.1 The default file used by the debugger.

To give you an idea of how the extension language looks like, look
at '/usr/local/lib/lispdebugger/lispsyntax' the source used to generate
the parser/transformer of the debugger delivered with this package.
Although the language used is not yet defined it should look familiar
to you.

4.3.2 Syntax of the language. 

The language is composed of the following elements :

comments Everything on a line after a ';' is considered a comment
  and is neglected.

white spaces Used as separators , the following is a white space :

* blanc

* newline

* tab

* return

definitions These are of the form : 

* symbol = expression

expressions Thes are of the form :

* symbol

* _symbol

* ~symbol

* #symbol

* @

* [ expression ... expression ]

* [ expression ... expression ]*

* {expression ... expression}

* {expression ...|....|...| expression... }*

* ( expression expresion ...)

* ``text''

symbols Any string of characters with the exception of white spaces
  , @ , _ ,~,#,(,),[,]

A source in our language is a text file containing definitions and
expressions.

4.3.3 Semantics of the language.

The syntax tells us what the welformed expressions and definitions
are in our language but it says nothing about their meaning, for this
we need a little bit of semantics. The best way to understand the
semantics of the language is the consumer/producent metaphor. When
a expression in our language is applied on a lisp expression two things
can happens :

1. The expression recognizes the lisp expression consuming part of it
  and producing another lisp expression.

2. The expression does not recognize the expression an it generates
  a throw, no lisp expression is produced.

Lets now put these ideas in practice on the different type of expressions
of our language. Lets P the list produced, E a expression in our language
and L the lisp expression on which we applies expressions in our language.
The parser/generator in the debugger will apply each expression on
a given lisp expression until it gets not a throw and the lisp expression
is fully consumed, the produced list is then the lisp expression with
debugcode added. If this sounds inefficient you are right this is
just a semantic explanation , our language is actually compiled to
become the parser/generator of the debugger which has the same effect
as our semantic explanation, but he does it in a more efficient way.

4.3.3.1 Symbol.

If we apply a 'symbol' on a lisp expression L=(e1 e2... en) or L=()
we have a throw if e1 is not equal to our 'symbol' or if L is the
empty list. If e1 is equal to our 'symbol' then we append P with the
symbol and L becomes (e2 ... en). 

Let P=() , L=(defun f (n) (princ n)) and E=defun then applying E on
L gives L=(f (n) (princ n)) and P becomes (defun).

Let P=(), L=(defun f(n) (princ n)) and E=let then applying E on L gives
a throw.

4.3.3.2 ``text''

If we apply ``...'' on a lisp expression L=(e1 e2 ... en) or L=() we
have a throw if L is empty or if e1 is not a string. In all other
cases L becomes (e2 ... en) and the first element e1 is added to P.
You can think of ``text'' as standing for any string.

4.3.3.3 _symbol

If we apply '_symbol' on a lisp expression L=(e1 e2.... en) or L=()
we have a throw if L is empty. In all other cases L becomes (e2 ...
en) and the first element e1 is added to P. You can think of _symbol
as standing for any list element which must not be changed.

Let P=() , L=(a b c d) and E=_sym then applying E on L gives L=(b c
d) , P=(a).

Let P=(),L=((a b) c d) and E=_sym then applying E on L gives L=(c d)
, P=((a b))

Let P=(),L=() and E=_sym then applying E on L gives a throw.

4.3.3.4 ~symbol

If we apply '~symbol' on a lisp expression L=(e1 e2 .. en) or L=()
we have a throw if L is empty or e1 is a list. In all other cases
L becomes (e2 ... en) and the first element e1 is added to P. Also
e1 is added to the lexical environment of the debugger. During executing
of the debugged code the system test if the variable is defined in
the lexical environment at the executionpoint and if it is so the
binding of the variable is saved. This means that you can refere to
the binding of this variable during debugging. You don't have to worry
when this variable can be referred (the system keeps track of this)
only indicate that this is defined to become a variable.

Let P=() , L=(a b c d) and E=^sym then applying E on L gives L=(b c
d) , P=(a).

Let P=(),L=((a b) c d) and E=^sym then applying E on L gives a throw.

Let P=(),L=() and E=^sym then applying E on L gives a throw.

4.3.3.5 #symbol

If we apply '#symbol' on a lisp expression L=(e1 e2 ... en) or L=()
we have a throw if L is empty. In all other cases L becomes (e2 ....
en) and the system tries to add debugging code to e1 before it is
added to P. The debugging code added makes that you can place a breakpoint
on e1, and see where e1 is located in the source. In some cases no
debugging code is added (if e1 is not a list or if e1 represents a
macro not recognized by the debugger). You can use # to indicate that
you should be able to set breakpoints.

Let P=() , L=(a b c d) and E=^sym then applying E on L gives L=(b c
d) , P=((add-debug-code a)).

Let P=(),L=() and E=^sym then applying E on L gives a throw.

4.3.3.6 @

We can always apply @ to L . It consumes nothing and it produces nothing
, its solely purpose is for its side effect, it allows you to set
a breakpoint on the whole expression where it is part of. Use this
in (defun ...) (let ...) just before the body of these functions.

4.3.3.7 [expression1 ... expressionn]

If we apply [expression1 ... expressionn] on a lisp expression L .
It will first do nothing , the expressions after [...] are first applied
on L if this gives a throw the system will first trying to apply expression1
then expression2 ... on L followed by the expressions after [...].
Consider this as a kind of optional syntax.

E=[a b] c , L=(a b c d) and P=() then applying E on L gives L=(d) ,
P=(a b c)

E=[a b] c , L=(c d) and P=() then applying E on L gives L=(d) , P=(c)

E=[a b] d , L= (c d) and P=() then applying E on L gives a throw

4.3.3.8 {e11...e1n|...| em1....emk}

If we apply {e11...e1n|...| em1....emk} on a lisp expression L . It
will first try to apply e11...e1n on L followed by all expressions
after { ... } if this causes a throw, it will try e21...e2l followed
by the rest ... . You can consider { ... } as a kind of or where you
want to try different alternatives.

E={a b} c , L=(a c d) and P=() then applying E on L gives L=(d),P=(a
c)

E={a b} c, L=(b c d) and P=() then applying E on L gives L=(d),P=(a
b)

E={a b} c, L=(c d) and P=() then applying E on L gives a throw

4.3.3.9 [expression1 | ... | expressionn]* , {expression1 ... expressionn}*

If we apply {...}* or [...]* on a lisp expression. We first try to
apply the expressions after { ...}* or [...]* , if this causes a throw
we try first {...} or [...] followed by the rest, if this fails then
we try {...}{...} or [...][...] and so on. To avoid infinite looping
we stop if one {..} or [...] gives a throw. Use this if you have more
then occurrence of the same elements.

E=[a b]* c, L=(a b a b c) and P=() then applying E on L gives L=(c)
, P=(a b a b)

4.3.3.10 (expression1 ... expressionn)

If we apply (expression1 ... expressionn) on L=(e1 ... en) or L=()
we get a throw if L is empty or if e1 is not a list . The result in
the other cases is the result of applying expression1 .... expressionn
on e1 where we start with a empty result list, this result list is
then added to the original list.If e1 is not consumed fully we have
also a throw.

E=(a b c) , L=((a b c) d) , P=() then applying E on L gives L=(d) ,
P=((a b c))

E=(a b c),L=((a b c d) d), P=() then applying E on L gives a throw.

4.3.3.11 definitions (name = expression)

A definition gives a name to a expression , applying this name has
the same effect as applying the expression. You can use definitions
to to define a expression ones and then use it many times. Definitions
can be used recursively , one can refer in a definition to itself.

4.3.4 Cavecats.

* When using recursive definitions make sure that you do not introduce
  infinite looping.

* Be carefull when using #,_,or ~and [],{}* and []* , the effect is
  not always what you want. Look at the following expression 

  * [{``string'' | (declare [_declaration])}] {#exp}* 

* This will when applied to (``Description'' (declare (string a)) (princ
  x)) apply with success {#exp}* to ``Description'' and (declare ..)
  and add debugcode to ``Description'' and (declare ...) which gives
  in the case of (declare ..) a syntax error in lisp. The way out
  of this problem is using some extra {}, our code becomes then something
  like :

    * {``string'' {#exp}* | ``string'' (declare [_declaration]) {#exp}*
      | 
       (declare [_declaration]) {#exp}* | (declare [_declaration]) ``string''
      {#exp}* | {#exp}*}

Porting of the debugger

The debugger is coded in such a way that it should be easy to port
it to another LISP implementation or even another Unix. This chapter
describes how such a port can be done.

5.0.1 Conditions.

The port will be the easiest if certain conditions are fullfilled:

1. The lisp adheres to ANSI LISP.

2. TCL/TK version 8.x (previous versions could work also but I have
  not tested them) is installed in the OS together with its libraries.
  In contrast to the previous version of the debugger , LISP must
  not support TCL/TK (except for GCL).

3. The OS and LISP supports sockets.

If (1) is not fullfilled then the best strategy to follow is to write
the missing ANSI functions used in the code of the debugger. In case
of failing of TCL/TK the whole interface must be rewritten using another
graphical library. If (3) is not fullfilled you should either extend
LISP with some C coding or choose another connection method to the
interface program. To help porting I will explain how the lisp system
talks with the interface.

5.0.2 Overview of the debugger.

To easy porting, the debugger is splitted in two components.

* The user interface , this is the GUI of the debugger and it is written
  in C using the TCL/TK system as a graphical library.

* A extension of lisp containing the code to parse and change sources,
  generate a new parser/generator and code called by the debugged
  functions.

The interaction between the interface and the lisp extensions is done
using sockets, because this is in my opinion the most supported IPC
in LISP and the OS. 

5.0.2.1 The GUI interface.

The C code is using the following C source files :

* interface.c , this contain the main body of the interface.

* hash.c , hash.h , this contains some extensions for hashing and list
  manipulation used in interface.c.

* tclinvoke.c , code to call TCL/TK in a more efficient way.

These three file must be compiled and linked together with the libraries
of TCL/TK to produce the executable 'interface'. This program can
be started in three ways , modifying the way how it talks to the lisp
system. For socket communication, the interface is always the server
and it is listening for connections of the lisp system to it. The
program 'interface' is started via the lisp system, who decides how
to start the interface. There are current three ways of dooing this.

1. 'interface', the communication takes place via a Unix socket with
  address '/tmp/lispdebugger' (this is the method used by CMUCL).

2. 'interface pid' , pid is the process-id of the parent of this progam
  (the lisp system). This pid is then used by 'interface' to send
  a signal SIGUSR1(=10) to the lisp system each time it has output
  for the lisp system and thus let the lisp system act on the input.
  The communication still takes place via Unix sockets with address
  '/tmp/lispdebugger' (this is the method used for GCL).

3. 'interface -port' , port is the port number to use , communication
  takes place via stream oriented tcp/INET sockets on the port indicated
  (this is used ACL and CLISP).

The communication from 'interface' to lisp is done by sending a lisp
command as a string followed by a return to the lispsystem (in case
of 2 after the send a signal is send to wakeup the lisp, in all other
cases the lisp system wakes up if there is input on their socket connection).
After sending a command, the 'interface' will not wait for a return.
The functions called from the 'interface' to the lispsystem are:

* (debug-open-file <filename> <compile-t-nil>) to add debugcode to
  the program (if needed compile it) and load the changed code in
  the lisp system.

* (debug-delete-watchpoint <tag>) delete the watchpoint identified
  by <tag>

* (debug-separate-watchpoint <tag>) display the watchpoint identified
  by <tag> in a separate window

* (debug-set-watchpoint <tag> <begin> <end> <source> <exp>) sets a
  watchpoint identified by <tag> , <begin>,<end>,<source> indentifies
  the place of the watchpoint in the source and <exp> is the exp to
  watch.

* (display-result-exp <exp>) evaluates <exp> and displays the result

* (display-watchpoints) display all current watchpoints

* (if-breakpoint <exp>) checks if we must stop in a watchpoint because
  of a conditional breakpoint

* (end-debug-eventloop) to end the eventloop at a debugpoint

* (return-result-exp <exp>) returns <exp> as the result of a evaluating
  a lisp form

* (step-back-in-time) causes the debugger to step back in time

* (step-forward-in-time) causes the debugger to step forwards in time

* (stop-interface) stops and cleans up the lisp system socket part
  of the interface.

Communication from the lisp system to the 'interface' takes place in
the following way :

* First a number followed by a blanco indicating the function to call
  in the 'interface' is send, this number implicit says also how to
  handle the arguments. The following functions must be implemented
  in the interface (function together with function-code and description).
  Positions are expressed in number of bytes from the beginning of
  the source.

  * 0=hightlight-source , arguments begin(integer),end(integer),type(string),color(string)
    , should highlight the source text in the indicated color from
    position=begin to position=end. type is used to make a distinction
    between selection.

  * 1=display-message, arguments message(string) , displays the message
    string.

  * 2=highlight-error, arguments begin(integer) highlights the line
    containing the position begin.

  * 3=set-possible-breakpoints , arguments source(string),begin(integer),end(integer)
    gives the debugger information about the begin and endpoints of
    breakpoints in the source.

  * 4=give-control-to-interface , arguments source(string),begin(integer),end(integer)
    called from the debugged code to allow the debugger to intervine.
    In the 'interface' give-control-interface blocks until the user
    steps or press continue ... .

  * 5-display-result, arguments nr-args(integer) = no of args to follow,
    the args next comes always in groups of five args: type(integer)(0=variable,1=exp,10=variable
    in separate window,11 exp in separate window),tag(string),exp(string),value(string).

  * 6=display-exp-in-interface, arguments exp(string),result(string)
    used to display the result of evaluating a expression.

  * 7=if-breakpoint, arguments none , check for conditional breakpoints.

  * 8=display-time-env, arguments begin(integer),end(integer),source(string)
    equivalent as give-control-to-interface but used in time traveling.

  * 9=setting, arguments var(string),value(string) used to set the
    options settings in the debugger.

  * 10=give-control-to-interface-after, arguments source(string),begin(integer),end(integer)
    called from the debugged code to allow the debugger to intervine.
    In the 'interface' give-control-interface blocks until the user
    steps or press continue ... .

  * 11=lisp-goes-to-debug called if evalling debugged code in lisp
    gives continuable error, the interface will then display this
    error and highlight the problem code.

* The arguments of the functions are send as follows :

  * integers, in printed format followed by a blanc character so they
    can be read with scanf.

  * strings, first the length of the string is send, then a blanc character
    followed by the string followed by a blanc character.

5.0.2.2 The lisp code.

The lispcode is splitted up in two parts :

* The main body of the debugger (debugger.lisp).

* The interface to the 'GUI interface' (cmucl.lisp,gcl.lisp,acl5.lisp,clisp.lisp).

If you want to port the debugger to another lisp, you can leave debugger.lisp
and write a new <lisp name>.lisp file to interface with the 'GUI interface'
If you use the current 'GUI interface' the following functions should
be implemented in this file:

* (start-interface) , to start 'interface' and make a socket connection.

* (stop-interface) , to clean up everything after 'interface' is stoppped.

* (send-command command &arg-list) , to send the command function code
  followed by the arguments to the interface.

* (process-incoming) see if there is input waiting on the socket read
  this in and evaluate it.

* A way of detecting input on the socket and then call process-incoming.

For examples I refere to cmucl.lisp,gcl.lisp,acl5.lisp and clisp.lisp.
