https://beyondloom.com/decker/lil.html
Lil: A Scripting Language
Lil is part of the technology that powers Decker, a multimedia
creative tool inspired by HyperCard. Decker uses Lil for adding
custom behavior to decks and the widgets within. Lil is designed to
be learned in layers, but it is a richly multi-paradigm language
which incorporates ideas from imperative, functional, declarative,
and vector-oriented languages.
on mode a do # line comment
r:()
each x in a
r[x]:1+r[x]
end
extract first key orderby value desc from r
end
mode[1,2,2,3,4,2,1]
[logo]
Lil's implementation needs to be fairly small, as an interpreter is
shipped with every standalone Decker document along with the rest of
the runtime. At the same time, Lil should be prepared for the
everyday needs of Decker users, with primitives to handle common
use-cases and a minimum of boilerplate.
The language therefore tries to thread the needle between a design
which is simple and a design which is ergonomically pleasant. There
is a small number of datatypes, features and primitives, which
compose in many useful ways. The query syntax generalizes to
manipulation of dictionaries, lists, and strings, and replaces many
individual primitive operations in other languages with a single
algorithmic framework. Simple things are easy, and complex things are
possible.
Types and Conversions
There are 7 types of value in Lil: numbers, strings, lists,
dictionaries, tables, functions, and interfaces.
* Numbers are floating-point values, possibly with a sign: 42 37.5
-29999.
* Strings are a sequence of characters, written enclosed in
double-quotes. The special characters backslash, double-quote,
and the newline character (\n) are preceded with a backslash
escape: "apple" "foo\nbar". The binary fuse and format operators
can both be used for concatenating strings.
* Lists are ordered sequences of values which can be indexed by
numbers. The empty list is written as (), and the unary operator
list constructs a single-element list from any value. Lists can
be joined with the , operator.
* Dictionaries are ordered sequences of pairings between keys and
values, both of which can be any type. Dictionaries can be made
by the binary operator dict, the extract statement, or a list can
be promoted to a dictionary by assigning at non-numeric keys.
Dictionaries can be unioned together with the , operator.
* Tables are a rectangular array of values for which every column
has a string as its key. Tables can be made with the unary
operator table or with the insert statement. The binary operators
, (append), join (natural join), and cross (cartesian join) offer
a selection of tabular joins.
* Functions have a name, take arguments, and return a result. They
are declared with on, and may be freely passed around or stored
in variables, but are otherwise totally opaque values.
* Interfaces are opaque dictionary-like values used to represent
system resources or Input/Output devices. Accessing or writing to
an interface may produce side-effects.
Lil has a soft, spongy, dynamic type system in which values do their
best to convert to a more relevant type as the need arises.
* When a number is required, strings are parsed, and lists or
dictionaries attempt to convert their first element. Otherwise,
the number 0 is used.
* When a string is required, numbers are formatted, and lists are
recursively converted to strings and joined. Otherwise, the empty
string is used.
* When a list is required, strings are treated as a list of their
characters (each a length-1 string), dictionaries are treated as
their value list, tables are treated as the list of their rows,
each a dictionary, and anything else (number, function,
interface) is enclosed in a length-1 list.
* When a dictionary is required, strings and lists become
dictionaries from their indices to their elements, tables become
dictionaries of their columns, and otherwise an empty dictionary
is used.
* When a table is required, lists become a single-column "value"
table, and dictionaries become two-column "key" and "value"
tables. Anything else is interpreted as if it were a list.
When indexing into values, we will uniformly refer to elements:
* An element of a number is itself.
* An element of a string is a character, which itself is a length-1
string.
* An element of a list is one of its items.
* An element of a dictionary is one of its values.
* An element of a table is one of its rows, as a dictionary.
* An element of an empty string, list, dictionary, or table is the
number 0.
The number zero and the empty string, list, or dictionary are all
falsey, and any other value (including any table, function, or
interface) is truthy.
Lil, the Imperative Language
Lil scripts are a sequence of expressions, including literal values,
references or assignments to variables, conditionals (if), each and
while loops, and function declarations (on).
Variable and function names may contain any alphanumeric characters
(as well as ? and _), but must not start with a number. Variable
assignments use the symbol :, read as "gets" or "becomes":
a: 23
longer_name: "a string"
If a value has never been explicitly placed in a variable, it
contains the number 0. You can access or assign to elements of a list
or dictionary by subscripting with brackets:
(11,22,33)[1]
"String"[2]
l[4]:2
d["fruit"]:"apple"
For convenience, indexing may also be performed with "dot notation",
where a name after the dot is treated as a string index. These
expressions are equivalent:
d["fruit"]:"cherry"
d.fruit:"cherry"
If you index a list with integers within range of its count it will
act like a list. If you index a list like a dictionary- with keys
that are not integers within range of its count- it will become a
dictionary.
b: 5 # b contains the number 5
b[0]: 5 # b becomes a length-1 list containing the number 5
c # c contains the number 0
c.fruit: "yes" # c now contains a dictionary with the key "fruit" and the value "yes"
When the leftmost part of an assignment expression is an expression-
not just a variable name- the value of that expression is amended and
a new value with appropriate changes is returned:
(11,22,33)[1]:44 # produces the list (11,44,33)
"Cat"[1]:"ive" # produces the string "Civet"
().baz:99 # amend the empty list at index "baz" (promoting it to a dictionary) with the value 99,
# producing the single-element dictionary {"baz":99}
(11,22).baz:33 # extend (11,22) into a dictionary, and bind "baz", giving {0:11,1:22,"baz":33}
Note the important distinctions between these similar-looking cases:
foo:11,22,33
foo[1]:44 # get the value in "foo", amend it at index 1, and store the amended list in "foo" again.
(foo)[1]:44 # the parenthesized portion is a subexpression which gets the value in "foo".
# amend it at index 1, and return the amended list, leaving "foo" unchanged.
(foo[1]):44 # this is not syntactically valid!
# assignment to a subexpression must have a dotted or bracketed index.
If you index with a dot followed immediately by another dot or
open-bracket it means "each element". Use this for drilling into
nested lists or dictionaries:
t:("AB","CD","EFG")
t[1] # "CD" # the 1st element of t
t.[1] # ("B","D","F") # the 1st element of each element of t
d.a.key:"apple"
d.b.key:"pear"
d # {"a":{"key":"apple"},"b":{"key":"pear"}}
d.a # {"key":"apple"}
d.a.key # "apple"
d..key # {"a":"apple","b":"pear"}
---------------------------------------------------------------------
Lil has a number of unary and binary operators- see appendix 1 and 2
for details. Many, like + and *, should be familiar from mathematics
or other programming langugages. The most important thing to remember
in Lil is that expressions have uniform operator precedence:
expressions are always carried out right-to-left, unless explicitly
parenthesized:
3*2+5 # 21
3*(2+5) # 21
(3*2)+5 # 11
You can make decisions with if. The keyword if is followed by a
conditional expression, one or more statements, and finally the
keyword end. The statements inside the if ... end will only be
executed if the conditional expression has a truthy value. You can
also optionally include an else keyword to divide the body of the if
into a truthy half and a falsey half.
if 1>2
"narp" # this doesn't happen
end
if 5
"yarp" # 5 is truthy, so this does happen!
end
if 5
"also yarp" # this happens...
else
"also narp" # but this does not!
end
You can iterate over the elements of a value with an each loop. The
keyword each is followed by zero or more variable names, the keyword
in, an expression giving the source to iterate over, one or more
statements forming the "body" of the loop, and finally the keyword
end. The source is always treated as a dictionary, and for each
iteration of the loop the variable names provided are assigned to the
value, key, and index of the current pairing in that dictionary. For
a list, key and index will naturally be identical. Here are a few
examples of each loops:
each x in 3,5,7
x
end
each val key in d
val,key
end
r:0
each x in range 10
r:r+x
end
You can also repeat a body of code an indefinite number of times with
a while loop. The keyword while is followed by a conditional
expression, one or more statements forming the "body" of the loop,
and finally the keyword end. The conditional expression is evaluated
before each iteration of the loop, and the loop stops if this ever
results in a falsey value. Here's an example:
a:5
while a>3
print[a] # prints 5, then 4, then stops.
a:a-1
end
To declare a function, use the keyword on followed by a name, zero or
more argument names, the keyword do, one or more statements
comprising the "body" of the function, and finally the keyword end.
To call a function, use its name and a set of bracketed expressions
corresponding to the arguments it takes. Extra arguments are ignored,
and missing arguments are bound as 0:
on pair x y do
x,y
end
pair[3 5] # returns a list of 3 and 5
pair[3] # returns a list of 3 and 0
pair[3 5 7] # returns a list of 3 and 5
pair[3,5] # careful: ',' joins items into lists- it is not an argument separator!
pair 3 5 # wrong: this is three separate statements, not a function call!
Lil, the Functional Language
As we've seen above, thinking about Lil as an everyday garden-variety
imperative language is perfectly sufficient for writing scripts. Some
characteristics, though, make it well-suited to the functional style
of programming, in which we aim to minimize mutation, compose our
program from functions which do not have "side-effects", and make use
of so-called "higher-order" functions- functions which take functions
as arguments.
Values in Lil are truly values. They have copy-on-write semantics: an
assignment to part of a value creates an entirely new value, leaving
any other references to the original value unchanged:
a:1,2,3
b:a
b[1]:5
show[a] # 1,2,3
show[b] # 1,5,3
Statements in Lil are always expressions. That is, they always return
a value and can be composed within larger expressions. Assignments
evaluate to the value being assigned. if returns the value of the
last statement in its taken half. (A missing else just means the
falsey half evaluates to 0.) while likewise evaluates to the last
statement on the last iteration of its body:
x:y:z # assign both x and y the value of z
a: if x>5 99 else 33 end # assign a to either 99 or 33
c: while b<100 b:b*2 end # assign c to 128
The each loop collects together the results of each iteration of the
loop and returns them. If the input was a dictionary, the output will
be a dictionary with the same keys. Otherwise, the output will be a
list. Some other languages call this operation map:
each x in 3,5,7 # (300,500,700)
x*100
end
t.foo:"one"
t.bar:"three"
each v in t # {"foo":3,"bar":5}
count v
end
You can pass functions to other functions by name:
on apply func do
func["two"]
end
on twice x do
x,x
end
apply[twice] # ("two","two")
Since on...end is an expression, you can also directly substitute it
into the call, just as with "anonymous" or "lambda" functions in
other languages.
apply[on thrice x do x,x,x end]
One advantage of functions always having a bound name is that there's
no special combinator plumbing required for anonymous recursion. If
the name never matters, consider using the name _ to signal your
intent.
Lil uses lexical scope: variables will resolve to the closest nested
binding available, and the local variables of a caller to a function
will not be visible or modified by the callee (unless the callee's
definition is nested in the caller):
global:333
on quux x do
local:99
x[77]
print[local] # 99
end
on zami x do
local:23
print[global,local,x] # 333,23,77
end
quux[zami]
Furthermore, functions close over variables in their lexical scope,
allowing for encapsulated "objects" with their own mutable state:
on counter x do
on inc do
x:x+1
end
end
a:counter[100]
b:counter[200]
print[a[]] # 101
print[a[]] # 102
print[b[]] # 201
print[a[]] # 103
print[x ] # 0
Whenever an assignment is carried out, if the variable name in
question has been assigned to in any surrounding lexical scope, the
assignment will update the closest definition. If the name has never
been assigned to before, a new local variable will be created at the
current scope. The on, each, and query (select, update, extract)
statements always create new local variables for their arguments,
loop variables, or columns, and can thus shadow (take precedence
over) outer local variables of the same name. When in doubt, use
unique names; it's much less confusing!
Finally, if you're mentally wed to the idea of expressing algorithms
recursively, you may be pleased to discover that Lil supports
tail-call elimination. If a recursive function calls itself (or
another function) as the final operation in the function, it will not
consume extra stack space:
on addrec x y do
if x>0
1+addrec[x-1 y] # addition happens after call returns. not tail-recursive!
else
y
end
end
on addtail x y do
if x>0
addtail[x-1 y+1] # tail-recursive call.
else
y
end
end
addtail[80000 5] # this won't blow the stack.
Lil, the Query Language
So far, we haven't looked much at tables. While one can operate on
tables by indexing them or iterate over their contents with an each
loop, the real power of tables comes from Lil's query syntax, which
resembles a simplified version of SQL. Unlike some of their SQL
counterparts, select, extract, update, and insert are all pure
operations which strictly return a new value as a result.
select is used for querying a table, and obtaining a table of
results. It is followed by a sequence of column expressions (possibly
with names), optionally a where, by, and/or orderby clause, each of
which should produce a list of results for each row, the keyword
from, and then finally an expression giving the source table. In the
context of these clauses- and when computing result columns- the
columns of the table will be bound as variables.
In a select, extract or update, clauses are carried out in the
following order:
* rows are filtered down to only those with a truthy result in the
where clause. (The implicit index column represents their
original row numbers.)
* rows are grouped by the values in the by clause, in order of
appearance. (The implicit gindex column represents each row's
original index within its group, and the implicit group column
represents the index of the group each row belongs to.)
* each group is sorted according to the values in the orderby
clause, either ascending (asc) or descending (desc). As with <
and >, numbers are sorted by value, and anything else is sorted
lexicographically by its string equivalent.
* for each group, result columns are computed.
* for each group, result columns are rectangularized: the group
will have as many result rows as the column of maximum count.
* rows are ungrouped: result rows across all groups are
concatenated, preserving their grouped order. (This may change
the overall order- see update if you want to preserve it.)
Given a simple table:
people.name:"Alice","Sam","Thomas","Sara","Walter"
people.age:25,28,40,34,43
people.job:"Developer","Sales","Developer","Developer","Accounting"
people:table people
Let's perform some selections:
select from people # select all columns, like "select *" in SQL
# +----------+-----+--------------+
# | name | age | job |
# +----------+-----+--------------+
# | "Alice" | 25 | "Developer" |
# | "Sam" | 28 | "Sales" |
# | "Thomas" | 40 | "Developer" |
# | "Sara" | 34 | "Developer" |
# | "Walter" | 43 | "Accounting" |
# +----------+-----+--------------+
select name from people # select a specific column
# +----------+
# | name |
# +----------+
# | "Alice" |
# | "Sam" |
# | "Thomas" |
# | "Sara" |
# | "Walter" |
# +----------+
select firstName:name dogYears:7*age from people # compute and rename columns
# +-----------+----------+
# | firstName | dogYears |
# +-----------+----------+
# | "Alice" | 175 |
# | "Sam" | 196 |
# | "Thomas" | 280 |
# | "Sara" | 238 |
# | "Walter" | 301 |
# +-----------+----------+
select name where ("S%m" parse name) from people # where takes a boolean (0/1) column
# +--------+
# | name |
# +--------+
# | "Sam" |
# | "Sara" |
# +--------+
select name index orderby name asc from people # orderby takes a column of values and "asc" or "desc"
# +----------+-------+
# | name | index |
# +----------+-------+
# | "Alice" | 0 |
# | "Sam" | 1 |
# | "Sara" | 3 |
# | "Thomas" | 2 |
# | "Walter" | 4 |
# +----------+-------+
select name job by job orderby name asc from people # sort names within each group
# +----------+--------------+
# | name | job |
# +----------+--------------+
# | "Alice" | "Developer" |
# | "Sara" | "Developer" |
# | "Thomas" | "Developer" |
# | "Sam" | "Sales" |
# | "Walter" | "Accounting" |
# +----------+--------------+
All result columns will be repeated to match the count of the longest
result. Thus, if all the new columns yield a non-listy result, the
group is effectively collapsed into a summary row:
select employed:(count name) job by job from people
# +----------+--------------+
# | employed | job |
# +----------+--------------+
# | 3 | "Developer" |
# | 3 | "Developer" |
# | 3 | "Developer" |
# | 1 | "Sales" |
# | 1 | "Accounting" |
# +----------+--------------+
select employed:(count name) job:(first job) by job from people
# +----------+--------------+
# | employed | job |
# +----------+--------------+
# | 3 | "Developer" |
# | 1 | "Sales" |
# | 1 | "Accounting" |
# +----------+--------------+
When computing columns, you're working with lists of elements, and
taking advantage of the fact that primitives like < and +
automatically "spread" to lists. If you want to call your own
functions- say, to average within a grouped column- write them to
accept a list like so:
on avg x do ((sum x) / count x) end
select job:(first job) avg_age:avg[age] by job from people
# +--------------+---------+
# | job | avg_age |
# +--------------+---------+
# | "Developer" | 33 |
# | "Sales" | 28 |
# | "Accounting" | 43 |
# +--------------+---------+
The same applies to any functions called in a where, by, or orderby
expression. The predefined aggregation functions sum, raze, min, and
max may come in handy!
---------------------------------------------------------------------
The update statement has the same syntax and clauses as select, but
behaves differently: results are merged with the original table, and
the original order of rows is preserved. orderby can be used with
update, but will only impact the inputs to column expressions, not
the order of the result rows.
update job:"Engineer" where job="Developer" from people
# +----------+-----+--------------+
# | name | age | job |
# +----------+-----+--------------+
# | "Alice" | 35 | "Engineer" |
# | "Sam" | 28 | "Sales" |
# | "Thomas" | 50 | "Engineer" |
# | "Sara" | 44 | "Engineer" |
# | "Walter" | 43 | "Accounting" |
# +----------+-----+--------------+
As with select you can compute new named columns- values will be
filled in with 0 for rows masked off by a where clause.
update manager:random[name] where job="Developer" from people
# +----------+-----+--------------+---------+
# | name | age | job | manager |
# +----------+-----+--------------+---------+
# | "Alice" | 25 | "Developer" | "Sara" |
# | "Sam" | 28 | "Sales" | 0 |
# | "Thomas" | 40 | "Developer" | "Sara" |
# | "Sara" | 34 | "Developer" | "Sara" |
# | "Walter" | 43 | "Accounting" | 0 |
# +----------+-----+--------------+---------+
---------------------------------------------------------------------
When mixing queries with other types of code, it may be very useful
to execute a query and get back simple lists, strings, or numbers.
The extract statement is another variation on select which unpacks
its result directly into a value, instead of a result column in a
table:
jobs:extract first job by job from people
# ("Developer","Sales","Accounting")
extract gives you access to a variety of useful operations for
tables, dictionaries, or even lists:
extract value orderby value asc from jobs # sort a list
# ("Accounting","Developer","Sales")
extract index orderby value asc from jobs # grade a list
# (2,0,1)
extract value orderby index desc from jobs # reverse a list
# ("Accounting","Sales","Developer")
extract list index by value from "ABBAAC" # group a list
# ((0,3,4),(1,2),(5))
extract list value by floor index/3 from "ABCDEFGHI" # partition a list
# (("A","B","C"),("D","E","F"),("G","H","I"))
extract first value by value from "ABBAAC" # distinct items in a list
# ("A","B","C")
If names are specified, all results are collected into a dictionary:
extract a:first age b:last age orderby age asc from people
# {"a":25,"b":43}
---------------------------------------------------------------------
The insert statement adds new rows to a table. It is followed by a
sequence of one or more column names followed by a colon and an
expression, the into keyword, and an expression evaluating to a
table.
insert name:"John" job:"Writer" age:32 into people
# +----------+-----+--------------+
# | name | age | job |
# +----------+-----+--------------+
# | "Alice" | 25 | "Developer" |
# | "Sam" | 28 | "Sales" |
# | "Thomas" | 40 | "Developer" |
# | "Sara" | 34 | "Developer" |
# | "Walter" | 43 | "Accounting" |
# | "John" | 32 | "Writer" |
# +----------+-----+--------------+
Missing columns will be filled in as 0. If any of the inserted values
are lists, the remaining columns will be spread to match:
insert name:("John","Eric") age:32 zodiac:("Taurus","Virgo") into people
# +----------+-----+--------------+----------+
# | name | age | job | zodiac |
# +----------+-----+--------------+----------+
# | "Alice" | 25 | "Developer" | 0 |
# | "Sam" | 28 | "Sales" | 0 |
# | "Thomas" | 40 | "Developer" | 0 |
# | "Sara" | 34 | "Developer" | 0 |
# | "Walter" | 43 | "Accounting" | 0 |
# | "John" | 32 | 0 | "Taurus" |
# | "Eric" | 32 | 0 | "Virgo" |
# +----------+-----+--------------+----------+
As a special case, if the value to insert into is a number, treat it
as an empty table. This is a convenient way to make a table from
scratch:
insert name:("John","Eric") age:32 zodiac:("Taurus","Virgo") into 0
# +--------+-----+----------+
# | name | age | zodiac |
# +--------+-----+----------+
# | "John" | 32 | "Taurus" |
# | "Eric" | 32 | "Virgo" |
# +--------+-----+----------+
---------------------------------------------------------------------
Lil also offers two basic joining operations: join (natural join),
and cross (cartesian/cross join):
jobs:insert job:"Sales","Developer","Accounting","Facilities" salary:85000,75000,60000,50000 into 0
# +--------------+--------+
# | job | salary |
# +--------------+--------+
# | "Sales" | 85000 |
# | "Developer" | 75000 |
# | "Accounting" | 60000 |
# | "Facilities" | 50000 |
# +--------------+--------+
people join jobs
# +----------+-----+--------------+--------+
# | name | age | job | salary |
# +----------+-----+--------------+--------+
# | "Alice" | 25 | "Developer" | 75000 |
# | "Sam" | 28 | "Sales" | 85000 |
# | "Thomas" | 40 | "Developer" | 75000 |
# | "Sara" | 34 | "Developer" | 75000 |
# | "Walter" | 43 | "Accounting" | 60000 |
# +----------+-----+--------------+--------+
guests:insert name:"Alice","Joan","Oscar","Thomas" into 0
# +----------+
# | name |
# +----------+
# | "Alice" |
# | "Joan" |
# | "Oscar" |
# | "Thomas" |
# +----------+
select a:name b:name_ where name < name_ from guests cross guests
# +---------+----------+
# | a | b |
# +---------+----------+
# | "Alice" | "Joan" |
# | "Alice" | "Oscar" |
# | "Joan" | "Oscar" |
# | "Alice" | "Thomas" |
# | "Joan" | "Thomas" |
# | "Oscar" | "Thomas" |
# +---------+----------+
For a row-wise join- concatenating tables with the same columns- you
can simply use ,, which has the same padding/widening behavior as
insert. You can likewise take or drop rows or columns from a table.
If the left argument to take is a list of numbers, it picks out those
rows very much like a generalization of select:
"name" take people
# +----------+
# | name |
# +----------+
# | "Alice" |
# | "Sam" |
# | "Thomas" |
# | "Sara" |
# | "Walter" |
# +----------+
("age","job") take people
# +-----+--------------+
# | age | job |
# +-----+--------------+
# | 25 | "Developer" |
# | 28 | "Sales" |
# | 40 | "Developer" |
# | 34 | "Developer" |
# | 43 | "Accounting" |
# +-----+--------------+
3 drop people
# +----------+-----+--------------+
# | name | age | job |
# +----------+-----+--------------+
# | "Sara" | 34 | "Developer" |
# | "Walter" | 43 | "Accounting" |
# +----------+-----+--------------+
(0,2,3) take people
# +----------+-----+-------------+
# | name | age | job |
# +----------+-----+-------------+
# | "Alice" | 25 | "Developer" |
# | "Thomas" | 40 | "Developer" |
# | "Sara" | 34 | "Developer" |
# +----------+-----+-------------+
The flip of a table transposes its data, promoting the first column
(or, if it exists, a column named key) from the original table to
column keys for the result:
expenses: insert kind:"tax","gas","power","food" jan:11,22,33,44 feb:55,66,77,88 into 0
# +---------+-----+-----+
# | kind | jan | feb |
# +---------+-----+-----+
# | "tax" | 11 | 55 |
# | "gas" | 22 | 66 |
# | "power" | 33 | 77 |
# | "food" | 44 | 88 |
# +---------+-----+-----+
flip expenses
# +-------+-----+-----+-------+------+
# | key | tax | gas | power | food |
# +-------+-----+-----+-------+------+
# | "jan" | 11 | 22 | 33 | 44 |
# | "feb" | 55 | 66 | 77 | 88 |
# +-------+-----+-----+-------+------+
The key column makes the flip of a table a reversible operation. If
you don't need the original keys, you can discard them with drop:
"key" drop flip expenses
# +-----+-----+-------+------+
# | tax | gas | power | food |
# +-----+-----+-------+------+
# | 11 | 22 | 33 | 44 |
# | 55 | 66 | 77 | 88 |
# +-----+-----+-------+------+
---------------------------------------------------------------------
In general, use select for narrowing down or summarizing tables,
update for making changes to a table while preserving or extending
its existing structure, extract for pulling data out of a table for
use elsewhere, and insert to make new tables or append to existing
ones.
It may also be helpful to think in terms of how the shape of output
tables relates to input tables for various table operations:
Query Statement Output Rows Output Columns
select <= input any number
update = input >= input
insert >= input = input
extract <= input n/a
join <= x * y < x + y
cross = x * y = x + y
flip x columns x rows
Lil, the Formatting Language
The parse and format primitives are used for breaking strings apart
into Lil values, and formatting Lil values into a customizable string
representation, respectively. Both take as their left argument a
format string with a concise notation for controlling these
processes. In many cases, these format strings are symmetrical: the
parse of the format under the same Format string will be an identity
operation, and vice versa:
f: "0x%04h" # a four-digit zero-padded lowercase hexadecimal number with an '0x' prefix
f format 123 # "0x007b"
f parse "0x007b" # 123
Format strings consist of a sequence of patterns and literals.
Patterns always begin with a % character, may contain several
optional flags, and end in an alphabetic character. Pattern flags
configure the details of how each pattern behaves, and may appear in
the structure %*-0N.DX, where N and D may be 1 or more digits 0-9,
and X is a pattern type. All flags are optional. If present, their
meanings are as follows:
* * this pattern will be validated, but will not produce output
values. When formatting, instead of consuming an input value,
these patterns will use the appropriate null value.
* - specifies left-justification for N, as decribed below. The
default is right-justification. The %r/%o patterns use - to
invert the set of valid characters.
* 0 specifies padding with 0 for N, as described below. The default
is padding with spaces.
* N indicates that the value will be padded to at least N
characters, using the justification and padding character
selected with - and 0.
* .D for %f/%c, the value should be formatted with exactly D
decimal places. The %r/%o patterns use D to indicate a number of
valid characters. For any other type, the formatted string will
be truncated at D characters, respecting justification.
Pattern types are as follows:
Type Null Parsed Formatted
% n/a literal % character. * is %
implied.
n n/a number of chars that have been nothing.
read.
m n/a matched? value is 1 iff the nothing.
format has matched so far.
value is 1 iff the format
z n/a matches and read the whole nothing.
input.
s "" string. read N chars or until any string, up to D chars.
next literal.
u "" uppercase string. just like s, any string, up to D chars,
but converts to uppercase. converted to uppercase.
l "" lowercase string. just like s, any string, up to D chars,
but converts to lowercase. converted to lowercase.
r "" repeat. 0 or more (or N) any string, exactly N chars
characters within a valid set. if specified.
o "" optional. 0 or 1 (or N) any string, exactly N chars
characters within a valid set. if specified.
a () ASCII. reads like s; value is list of ASCII ordinals
list of ASCII ordinals. converted to a string.
b 0 reads like s; value is 1 iff any value to true or false
first char is in tTyYx1. based on truthiness.
f 0.0 Lil float. allows any number of show D decimal places or
decimals. however many are needed.
c 0.0 currency. parses values like show D or 2 decimal places,
-$1.23 to Lil floats. like -$1.23.
i 0 signed integer. signed integer.
h 0 hexadecimal integer. parses format int as hexadecimal
lower- or uppercase. in lowercase.
H 0 hexadecimal integer. parses format int as hexadecimal
lower- or uppercase. in uppercase.
j 0 a JSON value. any value to a JSON string.
e 0 read ISO-8601 date-time into a format unix epoch int as
unix epoch int. ISO-8601 date-time.
p () read ISO-8601 date-time as a format dict as ISO-8601.
dictionary of time parts.
When parsing, each pattern will be matched against input in sequence,
consuming some number of input characters and producing output
values, and literals will be expected; if at any point a pattern or
literal fails to match against input, parsing will cease, and any
subsequent patterns in the format string will yield appropriate
"null" values. Thus, a given format string will always yield a fixed
number of results, no matter the input. In the case where there is
exactly one value-yielding pattern in the format string, parse output
will simply be that value instead of a list.
"%f %s %i" parse "12 apples" # (12,"apples",0)
"%f %ss" parse "12 apples" # (12,"apple")
("amount","noun") dict "%f %ss" parse "12 apples" # {"amount":12,"noun":"apple"}
"[%s]" parse "[something]" # "something"
The right argument to parse may be list of strings; in this case the
output will be a list of rows, where each row contains the values
from parsing one input string. The flip of this would give columns
instead. Here's an example of parsing a table of fixed-width records:
form: "%6s%6c%2i"
data: "apple $1.00 1\ncherry $0.3515\nbanana $0.75 2"
r: form parse "\n" split data
#(("apple ",1,1),("cherry",0.35,15),("banana",0.75,2))
table r
#+----------+------+----+
#| c0 | c1 | c2 |
#+----------+------+----+
#| "apple " | 1 | 1 |
#| "cherry" | 0.35 | 15 |
#| "banana" | 0.75 | 2 |
#+----------+------+----+
flip r
#(("apple ","cherry","banana"),(1,0.35,0.75),(1,15,2))
t: table ("name","price","amt") dict flip r
#+----------+-------+-----+
#| name | price | amt |
#+----------+-------+-----+
#| "apple " | 1 | 1 |
#| "cherry" | 0.35 | 15 |
#| "banana" | 0.75 | 2 |
#+----------+-------+-----+
("\n",form) format t # round-trip
#"apple $1.00 1\ncherry $0.3515\nbanana $0.75 2"
When formatting, literals will be included in the output string and
each pattern will control conversion of one value from the input
list, with a few exceptions as explained below. Missing arguments
will be interpreted as appropriate "null" values.
"%i,%a,%i" format 1,(list 65,66,67) # "1,ABC,0"
The left argument to format may be a list of strings; in this case it
is a series of alternating delimiters and format strings, with the
last item always being a format string. Each format string is
"pushed" one layer deeper into the right argument of format, and (if
present), the delimiter will be used to fuse these intermediate
strings, allowing a simple statement to recursively format a complex
structure:
() format 11,22 # (11,22) # (identity)
"%03i" format 11,22 # "011" # format the first item and discard unused(!)
(list "%03i") format 11,22 # ("011","022") # format each
(":","%03i") format 11,22 # "011:022" # fuse (format each)
("<%s>",":","%03i") format (list 11,22),(list 33) # ("<011:022>","<033>") # format each (fuse (format each))
("@","<%s>",":","%03i") format (list 11,22),(list 33) # "<011:022>@<033>" # fuse (format each (fuse (format each)))
Recursive formats will "explode" tables into lists of row-lists:
t: insert alpha:"one","two" beta:11,22 into 0
#+-------+------+
#| alpha | beta |
#+-------+------+
#| "one" | 11 |
#| "two" | 22 |
#+-------+------+
(list "%u - %i") format t
#("ONE - 11","TWO - 22")
If N is specified, the %s,%a and %b patterns will read up to N
characters of input when parsing. Otherwise, the next character in
the format string will be interpreted as a delimiter, and input
characters will be read until that delimiter is encountered. (Note
that if %s/%a/%b is immediately followed by a pattern, the delimiter
will be %!) If %s appears at the end of the format string, it will
simply read the remainder of the input string.
The %p pattern operates on a dictionary with numeric fields for year,
month, day, hour, minute, second:
"%p" parse "2021-02-03T04:05:58Z" # {"year":2021,"month":2,"day":3,"hour":4,"minute":5,"second":58}
"%p" format ().year:1984 # "1984-01-00T00:00:00Z"
The %n pattern can be used for progressive parsing. It also offers a
way of finding the first index of a given character in a string:
data: "one,two,three"
"%s,%n" parse data # ("one",4)
"%s,%n" parse 4 drop data # ("two",4)
"%*sA%n" parse "BBCABA" # the index of the first 'A' is 4
The %m and %z patterns can be used to disambiguate between failing to
match and successfully parsing a null-equivalent value. You can also
do some kinds of pattern matching:
"%i%m" parse "23" # (23,1) # successful parse
"%i%m" parse "0" # (0,1) # successful parse of 0
"%i%m" parse "orange" # (0,0) # mismatch, defaulted to 0
"exe%m" parse "foo.exe","execute" # (0,1) # prefix match
"exe%z" parse "execute","exe" # (0,1) # full match
The %r pattern is followed by one or more "valid" characters, the
count given by D (or 1 by default). This pattern matches and collects
input if and only if the input characters are within this set of
valid characters. The - flag inverts this behavior, such that the
pattern matches and collects only characters which are not in the
valid set. If N is specified, exactly N characters must be matched;
otherwise %r will accept zero or more valid characters. The %o
pattern is exactly like %r, but if N is unspecified it will accept
zero or one valid characters:
"%.2r01" parse "01110201" # "01110" # grab any leading number of binary digits
"%*.2r01%z" parse "010","012" # (1,0) # string consists only of binary digits?
"%r-%i" parse "----45" # ("----",45) # grab any prefix of minus signs
"%o-" parse "A","-A","--A" # ("","-","-") # grab at most one minus sign
"%*o-%i" parse "-45" # 45 # discard any sign, and read an unsigned int
"#%-r\n\n%s" parse "# comment\nA" # (" comment","A") # read a Lil-style line comment until a newline
The %j pattern can be used to format or parse data as JSON. When
formatting JSON, Lil dictionary keys will be cast to strings, and
anything other than a number, string, list, or dictionary will become
a JSON null:
"%j" format (11,22) dict (33,44) # "{\"11\":33,\"22\":44}"
"%j" format list 11,22 # "[11,22]"
"%j" format table 11,22 # "null"
When parsing JSON, the value true will become the number 1, and false
or null will become the number 0. This JSON parser is highly tolerant
and will among other things accept non-string JSON values as
dictionary keys, missing , and : delimiters, and some missing
trailing delimiters. Postel's Law, baby!
"%j" parse "[true,false,null,1]" # (1,0,0,1)
"%j" parse "{11:22,33:44" # {11:22,33:44}
Lil, the Vector Language
Lil has a number of features influenced by "Vector-oriented"
languages like APL, J, K, and Q. The most obvious is conforming, in
which a number of primitive operators like + and - can be applied
either to single numbers or entire lists. This functionality is
essential to how Lil manipulates columns within queries.
Let's start with a unary operator. Applied to a list, - "penetrates"
to each list element:
-(5) # -5
-(10,-35) # (-10,35)
With a binary operator, a non-list item will "spread" and be paired
with each element in a list. Given two lists, corresponding element
from each list are "paired up":
100+10 # 110
100+(10,20) # (110,120)
(100,200)+10 # (110,210)
(100,200)+(10,20) # (110,220)
The same pattern is carried out recursively; you can operate on
arbitrarily high-dimensional data this way. Sometimes you may still
need an each or flip to get your operands to line up the way you
want:
2*(list 1,2,3),(list 3,4)
# ((2,4,6),(6,8))
2 cross 3
# ((0,0),(1,0),(0,1),(1,1),(0,2),(1,2))
each x in 2 cross 3 do (27,19)+x end
# ((27,19),(28,19),(27,20),(28,20),(27,21),(28,21))
flip(27,19)+flip 2 cross 3
# ((27,19),(28,19),(27,20),(28,20),(27,21),(28,21))
Conforming is why Lil has two different equality operators: =
(equals) conforms, and ~ (match) does not. Consider these cases:
22=11,22,33 # (0,1,0)
22~11,22,33 # 0
(11,22,33)=11,22,33 # (1,1,1)
(11,22,33)~11,22,33 # 1
()=11,22 # ()
()~11,22 # 0
In many situations, = and ~ are equivalent. Prefer ~ when you don't
need conforming behavior; it signals your intent more clearly to a
reader, since it is easy to tell without context that the result will
be a single number.
Combining conforming operators with reducing operators like sum and
raze offers many elegant and direct solutions to problems. Compare
each of these approaches to counting how many times a value needle
can be found in a list haystack:
needle: "apple"
haystack: ("frog","apple","chicken","toadstool","apple","rice","fish")
c:0 each x in haystack if x~needle c:c+1 end end c # imperative
c:0 each x in haystack c:c+x~needle end c # imperative, without a conditional
extract count value where value=needle from haystack # query
count needle take haystack # functional (filter)
sum needle=haystack # vector-oriented (spread-conforming =)
Lil, the Decker Language
Lil contains a number of language features and builtins specifically
intended for interaction with the Decker environment.
The send statement is followed by the name of a function, and then a
set of arguments in brackets, just as if the function were being
called. Instead of calling the named function directly, send finds
the next closest binding (or the most recently shadowed binding, if
you like) for the name and calls that function. This is particularly
useful for event handlers which need to "bubble" events up to a more
general handler, like a card's link definition deferring to the
deck's link definition.
The send statement can also be handy to override built-in functions
like go[] with your own code, so that you have a chance to perform
some work before, after, or instead of their default behavior.
Defining these overrides in the deck's script will make them apply
for events triggered from any card or widget:
on go x do
# provide our own novel 'special name':
if x~"Random"
x:random[deck.cards]
end
# use 'send' to call the original definition:
send go[x]
end
An interface appears similar to a dictionary, but indexing or
assignment through an interface may have side-effects, and the values
in fields may change over time:
sys
"%e" format sys.now
"2021-11-12T02:00:59Z"
"%e" format sys.now
"2021-11-12T02:01:07Z"
time:sys
time.now
1636682495
Interfaces cannot be defined from Lil programs- they are furnished by
a host application like Decker. Consult Decker's manual for a
description of the interfaces you can use in your scripts. It is also
not possible to enumerate the keys of an interface; they may have
infinitely many keys, populated on the fly. Thus, the in operator
will always return 0 when an interface is its right argument.
Interfaces can be compared with ~ and = using reference equality and
concatenated into lists using , like any other datatype:
sys~sys
1
sys=(sys,sys,123)
(1,1,0)
All interfaces will accept the "type" index, which behaves like
typeof. This is handy if you ever have a list, dictionary, or table
containing interfaces:
sys.type
"system"
(sys,sys,sys)..type
("system","system","system")
Accessing an invalid index will return 0, and attempting to write to
an invalid (or read-only) index will return the expression to the
right, like any other indexed assignment:
sys.bogus
0
sys.bogus:123
123
Appendix 1: Unary Primitives
The unary arithmetic primitives - (negation), ! (logical not), floor,
cos, sin, tan, exp (the exponential function), ln (natural log), and
sqrt (square root) conform, and generalize to both lists and numbers.
The unary aggregation primitives sum, raze, min, and max take a list
and collapse it into a single result as if by combining every element
of the list with the binary primitives +, ,, & and |, respectively.
typeof gives the name of the type of the argument; one of the strings
{ "number", "string", "list", "dict", "table", "function" }, or, in
the case of an interface, the name of that interface type.
count gives the number of elements in a value. The count of a number
is always 1.
first and last extract the first or last elements of a value.
range of a number produces a list of the integers [0,x). Applied to
anything else, it will produce a list of the keys of its dictionary
equivalent.
list of anything produces a list of length 1 containing that value.
flip will transpose the elements of a list of lists- swap the x and y
axes. For example, flip (list 1,2,3),(list 4,5,6) is ((1,4),(2,5),
(3,6)). The result will always be rectangular: sublists that are too
short will be padded with 0, and any non-list values will be spread
to take up an entire column in the result. Applied to a table x, flip
will produce a new table using a column named key (or the first
column) as new column keys, and all other columns will become rows.
The original keys will become a column named key in the resulting
table. This operation can be useful for "pivoting" categorical data.
rows converts a table into a list of dictionaries, each representing
a row of the table.
cols converts a table into a dictionary of uniform-length lists, each
representing a column of the table.
table constructs a table. Given a list-of-dictionaries (as produced
by rows) or a dictionary, it will reconstitute a table, taking the
union of any dictionary keys as columns and extending any short
columns as with take. A list of lists will be interpreted as a list
of rows, padded to rectangularity with 0 if needed, using default
column names {c0, c1... cn}. A list containing non-dictionary and
non-list values will be converted into a table with a single value
column.
mag computes the magnitude of a vector, or the euclidean distance
between a point and the origin. If its argument is anything except a
number or list of numbers it will descend recursively through the
argument before computing a magnitude, similar to how other unary
primitives conform. For example, mag ((list 9,0),(list 3,4),(list
0,7)) is (9,5,7). The magnitude of a single number is its absolute
value.
heading computes the angle in radians from the origin toward a point.
This is similar to the atan2(y,x) quandrant-disambiguating arctangent
provided in many math libraries. Like mag, heading will descend
recursively through any non list-of-number arguments, so it can be
applied to points, lists of points, and so on.
unit computes a unit vector (with mag of 1) pointing toward an angle
in radians. This operation conforms. If the angle is between 0 and 2
pi and the original mag was 1, unit is the inverse of heading.
Appendix 2: Binary Primitives
The binary arithmetic and comparison primitives +, -, *, /, %
(modulus), ^ (exponentiation), <, >, =, & (minimum), and | (maximum)
conform, and generalize to both lists and numbers.
The modulus operator % takes its arguments in the opposite order of
common notation- the divisor is the left argument. Thus, 5 %
3,4,5,6,7 is (3,4,0,1,2). In common usage, this order will require
fewer parentheses given Lil's right-to-left precedence rule.
The <, > and = comparison operators always produce the number 1 or 0
as a result. If both operands are a number, they are compared
numerically. If either is an interface, they are equal only if both
are identical. Otherwise, the arguments are treated as strings and
compared lexicographically.
The & and | operator calculate the minimum or maximum of their
arguments. As a consequence, for the numbers 0 and 1 they are
equivalent to logical "AND" and logical "OR"- thus their notation.
Applied to strings, they compare values lexicographically in the same
fashion as < and >.
x ~ y is the match operator. It produces the number 1 if x and y are
identical values. Unlike =, match does not convert arguments or
automatically "spread" to list elements; x and y must have identical
types to begin with. This is particularly important if you want to
e.g. check whether an item is the empty list: ()=1,2 yields (), but
()~1,2 yields 0.
x , y is the concatenation operator. It is used for joining items
together into lists. If x is a dictionary, y will be converted to a
dictionary, and the operator will take the union of their key-value
mappings, preferring any bindings in y over x when both are present.
If applied to two tables, their rows will be concatenated, with any
missing columns supplied as 0. Note that using , on two strings will
result in a list of two strings, whereas using + will result in a
single string!
x @ y is spread-indexing: x is indexed with each element of y. For
example, (11,22,33) @ 0,1,0,1,0 is (11,22,11,22,11). With an
appropriate index y, this operator can be used to reorder, duplicate,
or filter elements of a list x. The expression x @ y is essentially
equivalent to each v k i in y (x[v k i]) end, and so it can also be
used as shorthand for any each loop that would otherwise simply be
applying a function to each element of its source.
x split y breaks a string y apart on any instance of the string x,
resulting in a list of strings.
x fuse y conversely combines the strings in the list y with the
string x, resulting in a string. Combining split and fuse can allow
one to replace instances of one substring with another.
x dict y constructs a dictionary from a list of keys x and a list of
values y.
x take y and x drop y are very general operators for filtering and
reshaping data. Their behaviors depend on the type of the left and
right arguments, as summarized in the table below. When x is a
number, take will repeat elements from y if it isn't long enough. For
non-numeric x values, take can be thought of as set intersection,
while drop can be thought of as set difference.
x y take drop
number string first x characters remove first x characters
from y from y
number list first x elements from remove first x elements from
y y
number dict first x key/value remove first x key/value
pairs from y pairs from y
number table first x rows from y remove first x rows from y
negative string last x characters remove last x characters from
number from y y
negative list last x elements from remove last x elements from y
number y
negative dict last x key/value remove last x key/value pairs
number pairs from y from y
negative table last x rows from y remove last x rows from y
number
non-number string only characters in x remove instances of
/list of y characters in x from y
non-number list only elements in x of remove instances of elements
/list y in x from y
non-number dict only keys in x of y remove keys in x from y
/list
string table only column x of y remove column x from y
string table only columns in x of remove columns in x from y
list y
number table only rows x of y, in remove rows in x from y
list order
x limit y returns up to x items from y. Equivalent to if x'|'='|'&'|'|'|','|'~'|'@'|'split'|'fuse'|'dict'|'take'|'drop'|
'in'|'join'|'cross'|'parse'|'format'|'unless'|'limit'
DIGIT := '0'|'1'|'2'|'3'|'4'|'5'|'6'|'7'|'8'|'9'
NUMBER := '-'? DIGIT+ '.'? | DIGIT* '.' DIGIT+
STRING := '"' (NON_ESC|'\\'|'\"'|'\n')* '"'
LITERAL := NUMBER | STRING | '(' ')'
NAME := (ALPHA|'_'|'?') (ALPHA|'_'|'?'|DIGIT)*
ITER := (( each' NAME* 'in' ) | 'while') EXPR* 'end'
ON := 'on' NAME+ 'do' EXPR* 'end'
IF := 'if' EXPR* ( 'else' EXPR* )? 'end'
QUERY := ('select'|'extract'|'update')((NAME ':')?EXPR)*('where'EXPR)?('by'EXPR)?('orderby'EXPR('asc'|'desc'))?'from' EXPR
INSERT := 'insert' (NAME ':' EXPR) 'into' EXPR
SEND := 'send' NAME '[' EXPR* ']'
INDEX := ('.' NAME? | '[' EXPR* ']')*
ACCESS := NAME INDEX ( ':' EXPR )?
TERM := LITERAL | ITER | ON | IF | QUERY | INSERT | SEND | '(' EXPR ')' | MONAD EXPR | ACCESS
EXPR := TERM ( INDEX ( ':' EXPR )? | DYAD EXPR )?
PROGRAM := EXPR*