https://turtleware.eu/posts/Common-Lisp-and-WebAssembly.html

TurtleWare
About
Blog
Mastodon
Fossil
Codeberg
Patreon
RSS feed

Common Lisp and WebAssembly

Tagged as lisp, webassembly

Written on 2025-11-28 by Daniel Kochmanski

Table of Contents

 1. Building ECL
 2. Building WECL
 3. Building user programs
 4. Extending ASDF
 5. Funding

Using Common Lisp in WASM enabled runtimes is a new frontier for the
Common Lisp ecosystem. In the previous post Using Common Lisp from
inside the Browser I've discussed how to embed Common Lisp scripts
directly on the website, discussed the foreign function interface to
JavaScript and SLIME port called LIME allowing the user to connect
with a local Emacs instance.

This post will serve as a tutorial that describes how to build WECL
and how to cross-compile programs to WASM runtime. Without further
ado, let's dig in.

Building ECL

To compile ECL targeting WASM we first build the host version and
then we use it to cross-compile it for the target architecture.

git clone https://gitlab.com/embeddable-common-lisp/ecl.git
cd ecl
export ECL_SRC=`pwd`
export ECL_HOST=${ECL_SRC}/ecl-host
./configure --prefix=${ECL_HOST} && make -j32 && make install

Currently ECL uses Emscripten SDK that implements required target
primitives like libc. In the meantime, I'm also porting ECL to WASI,
but it is not ready yet. In any case we need to install and activate
emsdk:

git clone https://github.com/emscripten-core/emsdk.git
pushd emsdk
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh
popd

Finally it is time to build the target version of ECL. A flag
--disable-shared is optional, but keep in mind that cross-compilation
of user programs is a new feature and it is still taking shape. Most
notably some nuances with compiling systems from .asd files may
differ depending on the flag used here.

make distclean # removes build/ directory
export ECL_WASM=${ECL_SRC}/ecl-wasm
export ECL_TO_RUN=${ECL_HOST}/bin/ecl
emconfigure ./configure --host=wasm32-unknown-emscripten --build=x86_64-pc-linux-gnu \
            --with-cross-config=${ECL_SRC}/src/util/wasm32-unknown-emscripten.cross_config \
            --prefix=${ECL_WASM} --disable-shared --with-tcp=no --with-cmp=no

emmake make -j32 && emmake make install

# some files need to be copied manually
cp build/bin/ecl.js build/bin/ecl.wasm ${ECL_WASM}

Running from a browser requires us to host the file. To spin Common
Lisp web server on the spot, we can use one of our scripts (that
assume that quicklisp is installed to download hunchentoot).

export WEBSERVER=${ECL_SRC}/src/util/webserver.lisp
${ECL_TO_RUN} --load $WEBSERVER
# After the server is loaded run:
# firefox localhost:8888/ecl-wasm/ecl.html

Running from node is more straightforward from the console
perspective, but there is one caveat: read operations are not
blocking, so if we try to run a default REPL we'll have many nested I
/O errors because stdin returns EOF. Running in batch mode works fine
though:

node ecl-wasm/ecl.js --eval '(format t "Hello world!~%")' --eval '(quit)'
warning: unsupported syscall: __syscall_prlimit64
Hello world!
program exited (with status: 0), but keepRuntimeAlive() is set (counter=0) due to an async operation, so halting execution but not exiting the runtime or preventing further async execution (you can use emscripten_force_exit, if you want to force a true shutdown)

The produced wasm is not suitable for running in other runtimes,
because Emscripten requires additional functions to emulate setjmp.
For example:

wasmedge ecl-wasm/ecl.wasm
[2025-11-21 13:34:54.943] [error] instantiation failed: unknown import, Code: 0x62
[2025-11-21 13:34:54.943] [error]     When linking module: "env" , function name: "invoke_iii"
[2025-11-21 13:34:54.943] [error]     At AST node: import description
[2025-11-21 13:34:54.943] [error]     This may be the import of host environment like JavaScript or Golang. Please check that you've registered the necessary host modules from the host programming language.
[2025-11-21 13:34:54.943] [error]     At AST node: import section
[2025-11-21 13:34:54.943] [error]     At AST node: module

Building WECL

The previous step allowed us to run vanilla ECL. Now we are going to
use artifacts created during the compilation to create an application
that skips boilerplate provided by vanilla Emscripten and includes
Common Lisp code for easier development - FFI to JavaScript,
windowing abstraction, support for <script type='common-lisp'>, Emacs
connectivity and in-browser REPL support.

First we need to clone the WECL repository:

fossil clone https://fossil.turtleware.eu/wecl
cd wecl

Then we need to copy over compilation artifacts and my SLIME fork (
pull request) to the Code directory:

pushd Code
cp -r ${ECL_WASM} wasm-ecl
git clone git@github.com:dkochmanski/slime.git
popd

Finally we can build and start the application:

./make.sh build
./make.sh serve

If you want to connect to Emacs, then open the file App/lime.el,
evaluate the buffer and call the function (lime-net-listen
"localhost" 8889). Then open a browser at http://localhost:8888/
slug.html and click "Connect". A new REPL should pop up in your Emacs
instance.

It is time to talk a bit about contents of the wecl repository and
how the instance is bootstrapped. These things are still under
development, so details may change in the future.

 1. Compile wecl.wasm and its loader wecl.js

We've already built the biggest part, that is ECL itself. Now we link
libecl.a, libeclgc.a and libeclgmp.a with the file Code/wecl.c that
calls cl_boot when the program is started. This is no different from
the ordinary embedding procedure of ECL.

The file wecl.c defines additionally supporting functions for
JavaScript interoperation that allow us to call JavaScript and
keeping track of shared objects. These functions are exported so that
they are available in CL env. Moreover it loads a few lisp files:

  * Code/packages.lisp: package where JS interop functions reside
  * Code/utilities.lisp: early utilities used in the codebase (i.e
    when-let)
  * Code/wecl.lisp: JS-FFI, object registry and a stream to wrap
    console.log
  * Code/jsapi/*.lisp: JS bindings (operators, classes, ...)
  * Code/script-loader.lisp: loading Common Lisp scripts directly in
    HTML

After that the function returns. It is the user responsibility to
start the program logic in one of scripts loaded by the the script
loader. There are a few examples of this:

  * main.html: loads a repl and another xterm console (external
    dependencies)
  * easy.html: showcase how to interleave JavaScript and Common Lisp
    in gadgets
  * slug.html: push button that connects to the lime.el instance on
    localhost

The only requirement for the website to use ECL is to include two
scripts in its header. boot.js configures the runtime loader and
wecl.js loads wasm file:

<!doctype html>
<html>
  <head>
    <title>Web Embeddable Common Lisp</title>
    <script type="text/javascript" src="boot.js"></script>
    <script type="text/javascript" src="wecl.js"></script>
  </head>
  <body>
    <script type="text/common-lisp">
      (loop for i from 0 below 3
            for p = (|createElement| "document" "p")
            do (setf (|innerText| p) (format nil "Hello world ~a!" i))
               (|appendChild| "document.body" p))
    </script>
  </body>
</html>

I've chosen to use unmodified names of JS operators in bindings to
make looking them up easier. One can use an utility lispify-name to
have lispy bindings:

(macrolet ((lispify-operator (name)
             `(defalias ,(lispify-name name) ,name))
           (lispify-accessor (name)
             (let ((lisp-name (lispify-name name)))
               `(progn
                  (defalias ,lisp-name ,name)
                  (defalias (setf ,lisp-name) (setf ,name))))))
  (lispify-operator |createElement|)    ;create-element
  (lispify-operator |appendChild|)      ;append-child
  (lispify-operator |removeChild|)      ;remove-child
  (lispify-operator |replaceChildren|)  ;replace-children
  (lispify-operator |addEventListener|) ;add-event-listener
  (lispify-accessor |innerText|)        ;inner-text
  (lispify-accessor |textContent|)      ;text-content
  (lispify-operator |setAttribute|)     ;set-attribute
  (lispify-operator |getAttribute|))    ;get-attribute

Note that scripts may be modified without recompiling WECL. On the
other hand files that are loaded at startup (along with swank source
code) are embedded in the wasm file. For now they are loaded at
startup, but they may be compiled in the future if there is such
need.

When using WECL in the browser, functions like compile-file and
compile are available and they defer compilation to the bytecodes
compiler. The bytecodes compiler in ECL is very fast, but produces
unoptimized bytecode because it is a one-pass compiler. When
performance matters, it is necessary to use compile on the host to an
object file or to a static library and link it against WECL in file
make.sh - recompilation of wecl.wasm is necessary.

Building user programs

Recently Marius Gerbershagen improved cross-compilation support for
user programs from the host implementation using the same toolchain
that builds ECL. Compiling files simple: use target-info.lisp file
installed along with the cross-compiled ECL as an argument to
with-compilation-unit:

;;; test-file-1.lisp
(in-package "CL-USER")
(defmacro twice (&body body) `(progn ,@body ,@body))

;;; test-file-1.lisp
(in-package "CL-USER")
(defun bam (x) (twice (format t "Hello world ~a~%" (incf x))))

(defvar *target*
  (c:read-target-info "/path/to/ecl-wasm/target-info.lsp"))

(with-compilation-unit (:target *target*)
  (compile-file "test-file-1.lisp" :system-p t :load t)
  (compile-file "test-file-2.lisp" :system-p t)
  (c:build-static-library "test-library"
                          :lisp-files '("test-file-1.o" "test-file-2.o")
                          :init-name "init_test"))

This will produce a file libtest-library.a. To use the library in
WECL we should include it in the emcc invocation in make.sh and call
the function init_test in Code/wecl.c before script-loader.lisp is
loaded:

/* Initialize your libraries here, so they can be used in user scripts. */
extern void init_test(cl_object);
ecl_init_module(NULL, init_test);

Note that we've passed the argument :load to compile-file - it
ensures that after the file is compiled, we load it (in our case -
its source code) using the target runtime *features* value. During
cross-compilation ECL includes also a feature :cross. Loading the
first file is necessary to define a macro that is used in the second
file. Now if we open REPL in the browser:

> #'lispify-name
#<bytecompiled-function LISPIFY-NAME 0x9f7690>
> #'cl-user::bam
#<compiled-function COMMON-LISP-USER::BAM 0x869d20>
> (cl-user::bam 3)
Hello world 4
Hello world 5

Extending ASDF

The approach for cross-compiling in the previous section is the API
provided by ECL. It may be a bit crude for everyday work, especially
when we work with a complex dependency tree. In this section we'll
write an extension to ASDF that allows us to compile entire system
with its dependencies into a static library.

First let's define a package and add configure variables:

(defpackage "ASDF-ECL/CC"
  (:use "CL" "ASDF")
  (:export "CROSS-COMPILE" "CROSS-COMPILE-PLAN" "CLEAR-CC-CACHE"))
(in-package "ASDF-ECL/CC")

(defvar *host-target*
  (c::get-target-info))

#+(or)
(defvar *wasm-target*
  (c:read-target-info "/path/to/ecl-wasm/target-info.lsp"))

(defparameter *cc-target* *host-target*)
(defparameter *cc-cache-dir* #P"/tmp/ecl-cc-cache/")

ASDF operates in two passes - first it computes the operation plan
and then it performs it. To help with specifying dependencies ASDF
provides five mixins:

  * DOWNWARD-OPERATION: before operating on the component, perform an
    operation on children - i.e loading the system requires loading
    all its components.

  * UPWARD-OPERATION: before operating on the component, perform an
    operation on parent - i.e invalidating the cache requires
    invalidating cache of parent.

  * SIDEWAY-OPERATION: before operating on the component, perform the
    operation on all component dependencies - i.e load components
    that we depend on

  * SELFWARD-OPERATION: before operating on the component, perform
    operations on itself - i.e compile the component before loading
    it

  * NON-PROPAGATING-OPERATION: a standalone operation with no
    dependencies

Cross-compilation requires us to produce object file from each source
file of the target system and its dependencies. We will achieve that
by defining two operations: cross-object-op for producing object
files from lisp source code and cross-compile-op for producing static
libraries from objects:

(defclass cross-object-op (downward-operation) ())

(defmethod downward-operation ((self cross-object-op))
  'cross-object-op)

;;; Ignore all files that are not CL-SOURCE-FILE.
(defmethod perform ((o cross-object-op) (c t)))

(defmethod perform ((o cross-object-op) (c cl-source-file))
  (let ((input-file (component-pathname c))
        (output-file (output-file o c)))
    (multiple-value-bind (output warnings-p failure-p)
        (compile-file input-file :system-p t :output-file output-file)
      (uiop:check-lisp-compile-results output warnings-p failure-p
                                       "~/asdf-action::format-action/"
                                       (list (cons o c))))))

(defclass cross-compile-op (sideway-operation downward-operation)
  ())

(defmethod perform ((self cross-compile-op) (c system))
  (let* ((system-name (primary-system-name c))
         (inputs (input-files self c))
         (output (output-file self c))
         (init-name (format nil "init_lib_~a"
                            (substitute #\_ nil system-name
                                        :test (lambda (x y)
                                                (declare (ignore x))
                                                (not (alpha-char-p y)))))))
    (c:build-static-library output :lisp-files inputs
                                   :init-name init-name)))

(defmethod sideway-operation ((self cross-compile-op))
  'cross-compile-op)

(defmethod downward-operation ((self cross-compile-op))
  'cross-object-op)

We can confirm that the plan is computed correctly by running it on a
system with many transient dependencies:

(defun debug-plan (system)
  (format *debug-io* "-- Plan for ~s -----------------~%" system)
  (map nil (lambda (a)
             (format *debug-io* "~24a: ~a~%" (car a) (cdr a)))
       (asdf::plan-actions
        (make-plan 'sequential-plan 'cross-compile-op system))))

(debug-plan "mcclim")

In Common Lisp the compilation of subsequent files often depends on
previous definitions. That means that we need to load files. Loading
files compiled for another architecture is not an option. Moreover:

  * some systems will have different dependencies based on features
  * code may behave differently depending on the evaluation
    environment
  * compilation may require either host or target semantics for
    cross-compilation

There is no general solution except from full target emulation or the
client code being fully aware that it is being cross compiled. That
said, surprisingly many Common Lisp programs can be cross-compiled
without many issues.

In any case we need to be able to load source code while it is being
compiled. Depending on the actual code we may want to specify the
host or the target features, load the source code directly or first
compile it, etc. To allow user choosing the load strategy we define
an operation cross-load-op:

(defparameter *cc-load-type* :minimal)
(defvar *cc-last-load* :minimal)

(defclass cross-load-op (non-propagating-operation) ())

(defmethod operation-done-p ((o cross-load-op) (c system))
  (and (component-loaded-p c)
       (eql *cc-last-load* *cc-load-type*)))

;;; :FORCE :ALL is excessive. We should store the compilation strategy flag as a
;;; compilation artifact and compare it with *CC-LOAD-TYPE*.
(defmethod perform ((o cross-load-op) (c system))
  (setf *cc-last-load* *cc-load-type*)
  (ecase *cc-load-type*
    (:emulate
     (error "Do you still believe in Santa Claus?"))
    (:default
     (operate 'load-op c))
    (:minimal
     (ext:install-bytecodes-compiler)
     (operate 'load-op c)
     (ext:install-c-compiler))
    (:ccmp-host
     (with-compilation-unit (:target *host-target*)
       (operate 'load-op c :force :all)))
    (:bcmp-host
     (with-compilation-unit (:target *host-target*)
       (ext:install-bytecodes-compiler)
       (operate 'load-op c :force :all)
       (ext:install-c-compiler)))
    (:bcmp-target
     (with-compilation-unit (:target *cc-target*)
       (ext:install-bytecodes-compiler)
       (operate 'load-op c :force :all)
       (ext:install-c-compiler)))
    (:load-host
     (with-compilation-unit (:target *host-target*)
       (operate 'load-source-op c :force :all)))
    (:load-target
     (with-compilation-unit (:target *cc-target*)
       (operate 'load-source-op c :force :all)))))

To estabilish a cross-compilation dynamic context suitable for ASDF
operations we'll define a new macro WITH-ASDF-COMPILATION-UNIT. It
modifies the cache directory, injects features that are commonly
expected by various systems, and configures the ECL compiler. That
macro is used while the

;;; KLUDGE some system definitions test that *FEATURES* contains this or that
;;; variant of :ASDF* and bark otherwise.
;;;
;;; KLUDGE systems may have DEFSYSTEM-DEPENDS-ON that causes LOAD-ASD to try to
;;; load the system -- we need to modify *LOAD-SYSTEM-OPERATION* for that. Not
;;; to be conflated with CROSS-LOAD-UP.
;;;
;;; KLUDGE We directly bind ASDF::*OUTPUT-TRANSLATIONS* because ASDF advertised
;;; API does not work.
(defmacro with-asdf-compilation-unit (() &body body)
  `(with-compilation-unit (:target *cc-target*)
     (flet ((cc-path ()
              (merge-pathnames "**/*.*"
                               (uiop:ensure-directory-pathname *cc-cache-dir*))))
       (let ((asdf::*output-translations* `(((t ,(cc-path)))))
             (*load-system-operation* 'load-source-op)
             (*features* (remove-duplicates
                          (list* :asdf :asdf2 :asdf3 :asdf3.1 *features*))))
         ,@body))))

Note that loading the system should happen in a different environment
than compiling it. Most notably we can't reuse the cache. That's why
cross-load-op must not be a dependency of cross-compile-op. Output
translations and features affect the planning phase, so we need
estabilish the environment over operate and not only perform. We will
also define functions for the user to invoke cross-compilation, to
show cross-compilation plan and to wipe the cache:

(defun cross-compile (system &rest args
                      &key cache-dir target load-type &allow-other-keys)
  (let ((*cc-cache-dir* (or cache-dir *cc-cache-dir*))
        (*cc-target* (or target *cc-target*))
        (*cc-load-type* (or load-type *cc-load-type*))
        (cc-operation (make-operation 'cross-compile-op)))
    (apply 'operate cc-operation system args)
    (with-asdf-compilation-unit () ;; ensure cache
      (output-file cc-operation system))))

(defun cross-compile-plan (system target)
  (format *debug-io* "-- Plan for ~s -----------------~%" system)
  (let ((*cc-target* target))
    (with-asdf-compilation-unit ()
      (map nil (lambda (a)
                 (format *debug-io* "~24a: ~a~%" (car a) (cdr a)))
           (asdf::plan-actions
            (make-plan 'sequential-plan 'cross-compile-op system))))))

(defun cross-compile-plan (system target)
  (format *debug-io* "-- Plan for ~s -----------------~%" system)
  (let ((*cc-target* target))
    (with-asdf-compilation-unit ()
      (map nil (lambda (a)
                 (format *debug-io* "~24a: ~a~%" (car a) (cdr a)))
           (asdf::plan-actions
            (make-plan 'sequential-plan 'cross-compile-op system))))))

(defun clear-cc-cache (&key (dir *cc-cache-dir*) (force nil))
  (uiop:delete-directory-tree
   dir
   :validate (or force (yes-or-no-p "Do you want to delete recursively ~S?" dir))
   :if-does-not-exist :ignore))

;;; CROSS-LOAD-OP happens inside the default environment, while the plan for
;;; cross-compilation should have already set the target features.

(defmethod operate ((self cross-compile-op) (c system) &rest args)
  (declare (ignore args))
  (unless (operation-done-p 'cross-load-op c)
    (operate 'cross-load-op c))
  (with-asdf-compilation-unit ()
    (call-next-method)))

Last but not least we need to specify input and output files for
operations. This will tie into the plan, so that compiled objects
will be reused. Computing input files for cross-compile-op is
admittedly hairy, because we need to visit all dependency systems and
collect their outputs too. Dependencies may take various forms, so we
need to normalize them.

(defmethod input-files ((o cross-object-op) (c cl-source-file))
  (list (component-pathname c)))

(defmethod output-files ((o cross-object-op) (c cl-source-file))
  (let ((input-file (component-pathname c)))
    (list (compile-file-pathname input-file :type :object))))

(defmethod input-files ((self cross-compile-op) (c system))
  (let ((visited (make-hash-table :test #'equal))
        (systems nil))
    (labels ((normalize-asdf-system (dep)
               (etypecase dep
                 ((or string symbol)
                  (setf dep (find-system dep)))
                 (system)
                 (cons
                  (ecase (car dep)
                    ;; *features* are bound here to the target.
                    (:feature
                     (destructuring-bind (feature depspec) (cdr dep)
                       (if (member feature *features*)
                           (setf dep (normalize-asdf-system depspec))
                           (setf dep nil))))
                    ;; INV if versions were incompatible, then CROSS-LOAD-OP would bark.
                    (:version
                     (destructuring-bind (depname version) (cdr dep)
                       (declare (ignore version))
                       (setf dep (normalize-asdf-system depname))))
                    ;; Ignore "require", these are used during system loading.
                    (:require))))
               dep)
             (rec (sys)
               (setf sys (normalize-asdf-system sys))
               (when (null sys)
                 (return-from rec))
               (unless (gethash sys visited)
                 (setf (gethash sys visited) t)
                 (push sys systems)
                 (map nil #'rec (component-sideway-dependencies sys)))))
      (rec c)
      (loop for sys in systems
            append (loop for sub in (asdf::sub-components sys :type 'cl-source-file)
                         collect (output-file 'cross-object-op sub))))))

(defmethod output-files ((self cross-compile-op) (c system))
  (let* ((path (component-pathname c))
         (file (make-pathname :name (primary-system-name c) :defaults path)))
    (list (compile-file-pathname file :type :static-library))))

At last we can cross compile ASDF systems. Let's give it a try:

ASDF-ECL/CC> (cross-compile-plan "flexi-streams" *wasm-target*)
-- Plan for "flexi-streams" -----------------
#<cross-object-op >     : #<cl-source-file "trivial-gray-streams" "package">
#<cross-object-op >     : #<cl-source-file "trivial-gray-streams" "streams">
#<cross-compile-op >    : #<system "trivial-gray-streams">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "packages">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "mapping">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "ascii">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "koi8-r">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "mac">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "iso-8859">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "enc-cn-tbl">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "code-pages">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "specials">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "util">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "conditions">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "external-format">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "length">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "encode">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "decode">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "in-memory">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "stream">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "output">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "input">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "io">
#<cross-object-op >     : #<cl-source-file "flexi-streams" "strings">
#<cross-compile-op >    : #<system "flexi-streams">
NIL
ASDF-ECL/CC> (cross-compile "flexi-streams" :target *wasm-target*)
;;; ...
#P"/tmp/ecl-cc-cache/libs/flexi-streams-20241012-git/libflexi-streams.a"

Note that libflexi-streams.a contains all objects from both libraries
flexi-streams and trivial-gray-streams. All artifacts are cached, so
if you remove an object or modify a file, then only necessary parts
will be recompiled.

All that is left is to include libflexi-streams.a in make.sh and put
the initialization form in wecl.c:

extern void init_lib_flexi_streams(cl_object);
ecl_init_module(NULL, init_lib_flexi_streams);.

This should suffice for the first iteration for cross-compiling
systems. Next steps of improvement would be:

  * compiling to static libraries (without dependencies)
  * compiling to shared libraries (with and without dependencies)
  * compiling to an executable (final wasm file)
  * target system emulation (for faithful correspondence between load
    and compile)

The code from this section may be found in wecl repository

Funding

This project is funded through NGI0 Commons Fund, a fund established
by NLnet with financial support from the European Commission's Next
Generation Internet program. Learn more at the NLnet project page.

NLnet foundation logo NGI Zero Logo