https://notes.eatonphil.com/practical-common-lisp-on-the-jvm.html
Home RSS
Subscribe
August 5, 2021
Practical? Common Lisp on the JVM: A quick intro to ABCL for modern
web apps
common lisplisparmed bear common lispjavajvm
In a ridiculous attempt to prove an internet wrong about the
practicality of Lisp (Common Lisp specifically), I tried to get a
simple (but realistic) web app running. After four days and a patch
to ABCL I got something working.
The code I had in mind would look something like this:
(let* ((port 8080)
(server (make-server port)))
(route server "GET" "/" (lambda (ctx) "My index!"))
(route server "GET" "/search"
(lambda (ctx)
(template "search.tmpl" '(("version" "0.1.0")
("results" ("cat" "dog" "mouse")))))))
And search.tmpl would be some Jinja-like text file:
Version {{ version }}
{% for item in results %}
{{ item }}
{% endfor %}
The source code for this post can be found on Github.
Picking a language, libraries
Armed Bear Common Lisp (ABCL) is the only Common Lisp implementation
I'm aware of that can hook into a major ecosystem of libraries like
the JVM or CLR has. In theory, this makes it a safe suggestion for
folks who want the stability and resources of the ecosystem even if
they aren't using its flagship language.
I wanted to use some micro web framework like Spark or Micronaut.
The problem with libraries like Micronaut (and Jersey) is that they
do a lot of dynamic inspection to figure out how to register
controllers and whatnot. This is certainly convenient for developers
using the library in Java. But it becomes an ordeal when you're
trying to use the library through a foreign function interface (FFI)
in another language. An example of this is if a framework scans all
files in a directory for a @GET annotation.
On the other hand, Spark had a seeming hard-requirement about
bringing in a Websocket library which caused some issues during
configuration. So I ended up going with Jooby and Netty (as the
underlying server).
Finally, I looked into a few Jinja-like template libraries and
settled on Pebble since Jinjava wouldn't load for me.
3rd-party jars and foreign function calls
So you've got your maven dependencies and ran mvn install. Your
pom.xml looks like this:
4.0.0
com.github.eatonphil
abcl-rest-api-hello-world
1
io.jooby
jooby
2.10.0
io.jooby
jooby-netty
2.10.0
io.pebbletemplates
pebble
3.1.5
ABCL has a package called abcl-asdf that helps you resolve
dependencies through Maven and your filesystem. We'll import it and a
package it depends on (abcl-contrib):
(require :abcl-contrib)
(require :abcl-asdf)
All our code will go into a single main.lisp file.
To import a specific package from Maven you call abcl-asdf:resolve
with a colon-separated string containing the Maven package group id
and artifact id. Then you pass that result to abcl-asdf:as-classpath
and pass that result to java:add-to-classpath.
It will look like this:
(setf imports '("io.jooby:jooby"
"io.jooby:jooby-netty"
"io.pebbletemplates:pebble"))
(loop for import in imports
do (java:add-to-classpath
(abcl-asdf:as-classpath (abcl-asdf:resolve import))))
Now you can call functions within these packages. If you want to call
a Java method using only builtins it looks like (jcall "method"
"com.organization.package.Class" object arg1 arg2 ... argN). If you
want to call a static Java method you use (jstatic ...) instead of
(jcall ...).
It seems that ABCL will automatically convert simple types from their
Lisp representation to Java but it will not turn a list into an
array. If a Java function requires an array you'll have to do that
explicitly with a function like (java:jnew-array-from-list
"java.lang.String" my-string-list).
When using the builtin Java FFI you always need to use the fully
qualified name for classes like java.lang.Object for Object or
java.util.Array for Array.
Alternatively you can (require :jss) to get access to a simpler
syntax for making Java calls. A method call looks like (#"method"
object arg1 arg2 ... argN). Creating a new instance of an object is
calling (jss:jnew 'className). When you use JSS you don't need to
fully qualify a class name unless there are more than one class with
the same name. For example to create a new Jooby application instance
we can call (jss:jnew 'Jooby). As long as the class can be found in
the class path JSS will resolve it.
Some real code
The real code will look similar to the pseudo-code at the top of this
article. We'll stub out the library-specific wrappers for rendering a
template and for registering a route.
Fumbling around the Jooby source code we see this snippet of Java:
* Server server = new Netty(); // or Jetty or Utow
*
* App app = new App();
*
* server.start(app);
*
* ...
*
* server.stop();
Netty comes from the jooby-netty artifact in the io.jooby group on
Maven. And App is some object that extends io.jooby.Jooby. Since
we're not using an OOP language though we're going to try avoiding
classes as much as possible. So we'll just create a new instance of
io.jooby.Jooby and add routes directly to it.
(defun template (filename context)
"")
(defun route (app method path handler)
nil)
(defun register-endpoints (app)
(route app "GET" "/"
(lambda (ctx) "An index!"))
(route app "GET" "/search"
(lambda (ctx)
(template "search.tmpl" `(("version" "1.0.0")
("results" ,(java:jarray-from-list '("cat" "dog" "mouse")))))))
(route app "GET" "/hello-world"
(lambda (ctx) "Hello world!")))
(let* ((port 8080)
(server (jss:new 'Netty))
(app (jss:new 'Jooby)))
(register-endpoints app)
(#"setOptions" server (#"setPort" (jss:new 'ServerOptions) port))
(#"start" server app)
(#"join" server))
Easy enough. Now we just need to implement route and template.
Implementing Java classes in ABCL
We are again not going the happy path with fancy Java syntax (which
is fine if you're using Java) like the Jooby documentation suggests.
Scouring the Jooby source code again it looks like we can call route
on the Jooby class with a method string, a path string, and an
instance of an object implementing the io.jooby.Route.Handler
interface.
Since this handler argument is an interface, we cannot cheat again by
creating an instance of it we'll have to actually create a new class
in Lisp that extends it. Thankfully there's only one method we need
to implement to satisfy this interface, apply. It accepts a
io.jooby.Context object and returns a java.lang.Object. The framework
then does introspection to figure out what exactly the object is and
if it needs to transform it into a string to be returned as an HTTP
response body.
To create a new class in ABCL we call (java:jnew-runtime-class
"classname" :interfaces '("an interface name") :methods '(("method
name 1" "return type" ("first parameter type" ...) (lambda (this arg1
...) body)))):
(defun route (app method path handler)
(#"route"
app
method
path
(jss:new (java:jnew-runtime-class
(substitute #\$ #\/ (substitute #\$ #\- path))
:interfaces '("io.jooby.Route$Handler")
:methods `(
("apply" "java.lang.Object" ("io.jooby.Context")
(lambda (this ctx)
(funcall ,handler ctx))))))))
One thing to note is that when referring to a subclass within a file
we need to address it with the io.jooby.Route$Handler syntax rather
than as you might refer to it in Java as io.jooby.Route.Handler. In
the latter case ABCL thinks Route is a package when in fact it's just
a class.
If you run this now with abcl --load main.lisp. It will work until
you hit an endpoint. The problem is how Jooby tries to figure out the
real type of the returned object.
The app will crash somewhere around here calling analyzer.returnType
(route.getHandle()).
In this case it tries to open and parse the (Java) source code of our
application to try to find the return type for this apply function.
That's a problem since our code isn't Java. Through trial and error I
realized we can trick Jooby/Java/somebody into figuring out the
correct return type by adding another implementation of apply that
returns a String to our class.
The full route code now looks like this:
(defun route (app method path handler)
(#"route"
app
method
path
(jss:new (java:jnew-runtime-class
(substitute #\$ #\/ (substitute #\$ #\- path))
:interfaces '("io.jooby.Route$Handler")
:methods `(
;; Need to define this one to make Jooby figure out the return type
;; Otherwise it tries to read "this file" which isn't a Java file so cannot be parsed
("apply" "java.lang.String" ("io.jooby.Context")
(lambda (this ctx) nil))
;; This one actually gets called
("apply" "java.lang.Object" ("io.jooby.Context")
(lambda (this ctx)
(funcall ,handler ctx))))))))
You may wonder, why keep the original method around? Well it's
because during reflection, ABCL says no such method that returns
String exists in the Handler interface. That's fair I guess.
Implementing the template
The Java example on the Pebble homepage is perfect:
PebbleEngine engine = new PebbleEngine.Builder().build();
PebbleTemplate compiledTemplate = engine.getTemplate("home.html");
Map context = new HashMap<>();
context.put("name", "Mitchell");
Writer writer = new StringWriter();
compiledTemplate.evaluate(writer, context);
String output = writer.toString();
We can easily translate this into Lisp:
(defun hashmap (alist)
(let ((map (jss:new 'HashMap)))
(loop for el in alist
do (#"put" map (car el) (cadr el)))
map))
(defun template (filename context-alist)
(let* ((ctx (hashmap context-alist))
(path (java:jstatic "of" "java.nio.file.Path" filename))
(file (#"readString" 'java.nio.file.Files path))
(engine (#"build" (jss:new 'PebbleEngine$Builder)))
(compiledTmpl (#"getTemplate" engine filename))
(writer (jss:new 'java.io.StringWriter)))
(#"evaluate" compiledTmpl writer ctx)
(#"toString" writer)))
But if you run this abcl --load main.lisp and hit this /search
endpoint, it will blow up saying "no such method" exists at the call
to Path.of(filename).
After digging around I saw it was because Path.of is a variadic
function.
And while there are examples of using variadic functions when the
function only has a single parameter like java.util.Arrays.asList(T
...), employing that same technique here continued to result in "no
such method":
(path (java:jstatic "of" "java.nio.file.Path" filename (jnew-array "java.lang.String" 0)))
Eventually I found an example of someone doing reflect/invoke on this
kind of a function call and tried this logic on a local copy of the
ABCL source code.
It worked. So I opened a pull request.
So the full working code for template is:
(defun template (filename context-alist)
(let* ((ctx (hashmap context-alist))
(path (java:jstatic "of" "java.nio.file.Path" filename (java:jnew-array "java.lang.String" 0)))
(file (#"readString" 'java.nio.file.Files path))
(engine (#"build" (jss:new 'PebbleEngine$Builder)))
(compiledTmpl (#"getTemplate" engine filename))
(writer (jss:new 'java.io.StringWriter)))
(#"evaluate" compiledTmpl writer ctx)
(#"toString" writer)))
And to get this diff running locally:
$ mkdir ~/vendor
$ cd ~/vendor
$ git clone https://github.com/eatonphil/abcl
$ cd abcl
$ git checkout pe/more-variadic
$ sudo {dnf/brew/apt} install ant maven
$ ant -f build.xml
And to run main.lisp using this diff:
$ ~/vendor/abcl/abcl --load main.lisp
And to hit the API:
$ curl localhost:8080/search
Version 1.0.0
cat
dog
mouse
$ curl localhost:8080/hello-world
Hello world!%
Phew! Easy peasy.
Next up
I'm porting this example to Kawa to see how it fares. Blog post to
come.
Feedback
As always, I'd love to hear from you with questions or ideas.
In a ridiculous attempt to prove an internet wrong about the
practicality of Lisp (Common Lisp specifically), I tried to get a
simple (but realistic) web app running. After four days and a
patch to ABCL I got something working.https://t.co/5UUWNR8Wnn
pic.twitter.com/cZsx32IlKD
-- Phil Eaton (@phil_eaton) August 5, 2021
Home RSS
Subscribe