https://borretti.me/article/why-lisp-syntax-works

  * Home
  * 
  * Articles

Why Lisp Syntax Works

Lisp's unusual syntax is connected to its expressive power. How? Not
because of "homoiconicity", a word that has no meaning but leaves
people somewhat impressed, because it sounds fanciful and
mathematical. It's because of uniformity.

Uniform

Language syntax exists on a spectrum from uniform to specific. Lisp
(and XML) have a uniform syntax^1. The same syntactic construction of

(<operator> <arg1> <arg2> ... <argn>)

Is used for:

 1. Expressions.
 2. Expressions that would be statements in other languages (i.e.
    control flow).
 3. Special operators like if and let.
 4. Declarations, like function or class declarations.

This means that macros apply everywhere. The exact same defmacro
facility lets you write macros that operate on expressions, special
operators, and declarations.

You can write macros that make writing expressions easier, like
infix-math:

($ (x * 2) / (x * 3))

;; expands to (/ (* x 2) (* x 3))

You can write macros that operate on would-be statements, like for:

(for:for ((li in (list 1 2 3 4))
          (vi across #(a b c d)))
  (format T "~&~a ~a" li vi))

;; expands to
;;
;; (loop for li in (list 1 2 3 4)
;;       for vi across #(a b c d)
;;       do (format T "~&~a ~a" li vi))

You can write macros that operate on declarations, like deftable in
crane:

(deftable user ()
  (name :type text :uniquep t :nullp nil)
  (age :type integer :nullp t :initform 18)
  (description :type text))

;; expands to a whole bunch of ORM stuff

And macros can be defined succinctly, e.g.:

(defmacro if-let ((var cond) &body consequent alternate)
  `(let ((,var ,cond))
     (if ,var
       ,consequent
       ,alternate)))

Defining a macro does not require some ceremonious process of writing
an AST-manipulating program, registering it with the build system or
whatever, it can be done inline, in the source. The language is
extensible from within, without requiring you to string together some
build-time Rube Goldberg preprocessing pipeline.

And macros are Lisp code, so they can use whatever toolkit of list
and tree-manipulation code you bring in. You can even define
lexically-scoped macros.

Syntactic uniformity also means there is a lot less room for syntax
bikeshedding. Really the only thing you can bikeshed is the naming
convention for identifiers, and how S-expressions should be indented.

Common Lisp syntax works because of the conjunction of:

 1. Syntactic uniformity, allowing macros to be applied everywhere.
 2. Turing-complete macros that can execute arbitrary code at
    compile-time, and leverage libraries.
 3. An image-based development model that blurs the lines between
    compile-time and run-time.

Specific

At the other end of the spectrum you have languages with a highly
specific syntax. The best example is plain old fashioned SQL. This is
the grammar for the humble ALTER TABLE statement in Postgres:

ALTER TABLE [ IF EXISTS ] [ ONLY ] name [ * ]
    action [, ... ]
ALTER TABLE [ IF EXISTS ] [ ONLY ] name [ * ]
    RENAME [ COLUMN ] column_name TO new_column_name
ALTER TABLE [ IF EXISTS ] [ ONLY ] name [ * ]
    RENAME CONSTRAINT constraint_name TO new_constraint_name
ALTER TABLE [ IF EXISTS ] name
    RENAME TO new_name
ALTER TABLE [ IF EXISTS ] name
    SET SCHEMA new_schema
ALTER TABLE ALL IN TABLESPACE name [ OWNED BY role_name [, ... ] ]
    SET TABLESPACE new_tablespace [ NOWAIT ]
ALTER TABLE [ IF EXISTS ] name
    ATTACH PARTITION partition_name { FOR VALUES partition_bound_spec | DEFAULT }
ALTER TABLE [ IF EXISTS ] name
    DETACH PARTITION partition_name [ CONCURRENTLY | FINALIZE ]

where action is one of:

    ADD [ COLUMN ] [ IF NOT EXISTS ] column_name data_type [ COLLATE collation ] [ column_constraint [ ... ] ]
    DROP [ COLUMN ] [ IF EXISTS ] column_name [ RESTRICT | CASCADE ]
    ALTER [ COLUMN ] column_name [ SET DATA ] TYPE data_type [ COLLATE collation ] [ USING expression ]
    ALTER [ COLUMN ] column_name SET DEFAULT expression
    ALTER [ COLUMN ] column_name DROP DEFAULT
    ALTER [ COLUMN ] column_name { SET | DROP } NOT NULL
    ALTER [ COLUMN ] column_name DROP EXPRESSION [ IF EXISTS ]
    ALTER [ COLUMN ] column_name ADD GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( sequence_options ) ]
    ALTER [ COLUMN ] column_name { SET GENERATED { ALWAYS | BY DEFAULT } | SET sequence_option | RESTART [ [ WITH ] restart ] } [...]
    ALTER [ COLUMN ] column_name DROP IDENTITY [ IF EXISTS ]
    ALTER [ COLUMN ] column_name SET STATISTICS integer
    ALTER [ COLUMN ] column_name SET ( attribute_option = value [, ... ] )
    ALTER [ COLUMN ] column_name RESET ( attribute_option [, ... ] )
    ALTER [ COLUMN ] column_name SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
    ALTER [ COLUMN ] column_name SET COMPRESSION compression_method
    ADD table_constraint [ NOT VALID ]
    ADD table_constraint_using_index
    ALTER CONSTRAINT constraint_name [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
    VALIDATE CONSTRAINT constraint_name
    DROP CONSTRAINT [ IF EXISTS ]  constraint_name [ RESTRICT | CASCADE ]
    DISABLE TRIGGER [ trigger_name | ALL | USER ]
    ENABLE TRIGGER [ trigger_name | ALL | USER ]
    ENABLE REPLICA TRIGGER trigger_name
    ENABLE ALWAYS TRIGGER trigger_name
    DISABLE RULE rewrite_rule_name
    ENABLE RULE rewrite_rule_name
    ENABLE REPLICA RULE rewrite_rule_name
    ENABLE ALWAYS RULE rewrite_rule_name
    DISABLE ROW LEVEL SECURITY
    ENABLE ROW LEVEL SECURITY
    FORCE ROW LEVEL SECURITY
    NO FORCE ROW LEVEL SECURITY
    CLUSTER ON index_name
    SET WITHOUT CLUSTER
    SET WITHOUT OIDS
    SET ACCESS METHOD new_access_method
    SET TABLESPACE new_tablespace
    SET { LOGGED | UNLOGGED }
    SET ( storage_parameter [= value] [, ... ] )
    RESET ( storage_parameter [, ... ] )
    INHERIT parent_table
    NO INHERIT parent_table
    OF type_name
    NOT OF
    OWNER TO { new_owner | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
    REPLICA IDENTITY { DEFAULT | USING INDEX index_name | FULL | NOTHING }

That's half the grammar. I trimmed the second half because the point
is made.

SQL syntax isn't composable, the way Lisp is. Composability is about
having a large number of distinct operators with a small number of
expression holes, while SQL is all about having a small number of
operators with fifty different syntactic variants.

And you can't write macros for SQL. You only options are C-style
text-manipulation macros, cleanly staged and separated from the SQL
itself, or writing a program that parses the SQL into an AST (good
luck) and transforms it at the AST level.

The Mean

Most languages inhabit the middle ground. Generally, syntax at the
same level will have more or less the same form. Declarations all
look kind of the same, they start with a keyword (struct, fn, union)
and use brackets to contain their inner declarations (record fields,
function bodies). Statements may all end in semicolons. That kind of
thing.

But for most languages, writing even a simple template
match-and-replace macro system is hard because the syntax is too
specific and irregular. I tried it with C and the results were mixed.

So for most languages, the only way to do macros is typed
AST-manipulation code, like OCaml's PPX.

I maintain that Common Lisp is the gold standard of DX for macros,
but it works only because Common Lisp exists at a very narrow
conjunction of tradeoffs. In particular, image-based development is a
rarity nowadays, a Galapagos island feature that is undesirable in
many contexts, but it's the thing that makes it possible to have
Turing-complete macros that are defined in the same place as the
code, without needing to involve a build system.

And languages that don't share this evolutionary niche should simply
do away with macros. The complexity-benefit tradeoff is not worth it.

Footnotes

 1. Regular would be a better adjective, but that confuses it with
    regular languages. -

(c) 2014 - 2023 Fernando Borretti. Subscribe through RSS.