--- layout: ../Site.layout.js --- # `:screwlisps-knowledge/tangle` common lisp asdf package markdown tangling and lisp pathnames Here I exhibit a simple use of [`cl-series`](https://gitlab.common-lisp.net/rtoy/cl-series/\-/wikis/Series-User's-Guide)' [`scan-file`](https://gitlab.common-lisp.net/rtoy/cl-series/\-/wikis/Series-User's-Guide#scan-file-function) and [`collect-file`](https://gitlab.common-lisp.net/rtoy/cl-series/\-/wikis/Series-User's-Guide#collect-file-function) to tangle a markdown file into a lisp package in a specified lisp system. Further, I add a system stubbing function which uses [lisp `pathname`](https://www.lispworks.com/documentation/HyperSpec/Body/19_b.htm)s, which are a tricky topic. My motivation was to at least in common lisp reconcile my small-web markdown and lisp systems, by defining my lisp systems as tangles of [my](https://codeberg.org/tfw/screwlisps-kitten/) [small-web markdown](https://kitten.small-web.org/tutorials/markdown/). Edrx showed me that something like this could be done by modifying [`emacs` `eev`](https://anggtwu.net#eev)'s `code-c-d`, but, well, I know one hundred percent how what I just wrote here works. It is a little hard to remember that common-lisp pathname stuff, as Kent fairly frequently says: it was written for back when there was more than one style of filesystem. However, after thinking a bit about it, I am fairly happy with it. the gist is that I have 1. Always used [`make-pathname`](https://www.lispworks.com/documentation/HyperSpec/Body/f_mk_pn.htm). 1. Represented unknown paths as [relative and wild (`(:relative :wild)`)](https://www.lispworks.com/documentation/HyperSpec/Body/26_glo_v.htm#valid_pathname_directory). 1. Made a new pathname when you know more of the path. Wild pathnames can have more advanced uses, but this is not a bad idiom on modern systems anyway I think. The modern string convention is I think, more primitive. # eev setup This is a block comment, prior to the required package specification. Elsewhere I have talked about my focus on [embeddable common lisp](https://gitlab.com/embeddable-common-lisp/ecl). ``` #| ;; Allowing, the quicklisp links are for sbcl and sly. Still, it gives an idea.  (find-quicklisp-links "series")  (setq inferior-lisp-program "ecl")  (slime)  (setq eepitch-buffer-name "*slime-repl ECL*") (require :series) |# ``` # :screwlisps-knowledge/tangle package definition This *must* be the first lisp expression in the file (since from our [`:package-inferred-system` useage](https://github.com/fare/asdf/blob/master/doc/best_practices.md#package_inferred), experientially files won't exist but packages do). ``` (uiop:define-package :screwlisps-knowledge/tangle (:import-from :series install) (:mix :asdf :uiop :cl) (:export :tangle) (:nicknames :sk/tangle :screwniverse/tangle)) (in-package :sk/tangle) (eval-when (:compile-toplevel :execute) (install)) ``` For those who aren't used to common-lisp, asdf, and uiop: ## [ASDF](https://asdf.common-lisp.dev/asdf.html) `:package-inferred-system` details This is in my view a modern residential source filesystem in the sense that there is no "file" - only packages, where packages are inside systems. In order to make this work, each source file created in a system starts with either `defpackage` or [`uiop:define-package`](https://asdf.common-lisp.dev/uiop.html#index-define_002dpackage), and the package generally needs to be named `:system/dir1/dir2/dir3/filename` (downcased). Then, in the absense of a `(:mix ..)`, `(:mix :cl)` is assumed if I recall correctly, however I specifically `import` `install` from `series`. In the new package=file, at `compile-toplevel`, I call cl-series' `install` (which temporarily interns series' exported symbols into the current or specified package). ## Interactively muddling through it once ### Setup a test input and output file Let's do this in a comment in the file. In some sense, it is documentation of what we did. Let's also use the [common lisp formatted output](https://www.lispworks.com/documentation/HyperSpec/Body/22_c.htm) [`~@\newline` directive](https://www.lispworks.com/documentation/HyperSpec/Body/22_cic.htm) which will let us use indented `'''`s safely. ``` (with-open-file (*standard-output* #p"/tmp/test-tangle.md" :direction :output :if-does-not-exist :create) (format *standard-output* "# An MD file~@ having~@ ```~@ (lisp code)~@ ```~@ blocks~@ ## Such as~@ ```~@ ;; this.~@ ```~@ ``` but not this.~@ .")) ``` ### Let's read the codes of that using cl-series cl-Series provides working fakes of its static-analysis-generated tight, lazy, efficient functionality. However, the fakes are not pure common lisp. But you can use them to explore interactively in series' declarative style, and see what-you're-working-with while you're working on it, as [Hairylarry for example mentioned](https://gamerplus.org/@hairylarry/114760934821459035) doing. ``` (defparameter *filein* (scan-file #p"/tmp/test-tangle.md" #'read-line)) *filein* ``` Using eev-mode, on my right side I see: ``` #| SK/TANGLE> (defparameter *filein* (scan-file #p"/tmp/test-tangle.md" #'read-line)) *FILEIN* SK/TANGLE> *filein* #Z("# An MD file " "having " "```" "(lisp code)" "```" "blocks" "## Such as" "```" ";; this." "```" ".") SK/TANGLE> |# ``` I guess, I would like to know where lines starting with `'''` are. ``` (defparameter *codeblock-toggles* (#Mstring= (series "```") *filein*)) *codeblock-toggles* ``` Series' `series` creates an infinite series of whatever its arguement is, which I compare to the finite series of lines [`read-line`](https://www.lispworks.com/documentation/HyperSpec/Body/f_rd_lin.htm)ed from the file. We don't have to personally manage the end of file stuff. Outputs: ``` #| SK/TANGLE> (defparameter *codeblock-toggles* (#Mstring= (series "```") *filein*)) *CODEBLOCK-TOGGLES* SK/TANGLE> *codeblock-toggles* #Z(NIL NIL T NIL T NIL NIL T NIL T NIL) SK/TANGLE> |# ``` Looking good. Now here we come to an annoying-ish point for my personal intuitions about series. Sometimes we need to use a toggle as I am about to, because we-don't-know-what-happens-in-the-future. This is quite different to eagerly evaluated code we may be used to. ``` (defparameter *blocksp* (let* ((in-block nil)) (mapping ((maybe-latch *codeblock-toggles*) (line *filein*)) (if in-block (if maybe-latch (setq in-block (not in-block)) line) (and maybe-latch (setq in-block (not in-block)) nil))))) ``` Well, let's do an [assert](https://www.lispworks.com/documentation/HyperSpec/Body/m_assert.htm)ion. ``` (assert (equal "(lisp code)" (collect-nth 3 *blocksp*)) nil "~a should be \"(lisp code)\" in the test example." (collect-nth 3 *blocks*)) ``` I guess we just want the lines, and [`write-line`](https://www.lispworks.com/documentation/HyperSpec/Body/f_wr_stg.htm)ed to a file. ``` (collect-file "/tmp/test-tangle.lisp" (choose *blocksp* *blocksp*) 'write-line) ``` # The [tangle](https://www.cs.tufts.edu/~nr/cs257/archive/literate-programming/01-knuth-lp.pdf) function Let's just trivially collect our test assert case into a function. ``` (defun tangle (md-path system &aux (fileout (make-pathname :name (pathname-name md-path) :type "lisp" :directory (pathname-directory (system-source-directory system))))) (let* ((filein (scan-file md-path #'read-line)) (codeblock-toggles (#Mstring= (series "```") filein)) (blocksp (let ((in-block nil)) (mapping ((maybe-latch codeblock-toggles) (line filein)) (if in-block (if maybe-latch (setq in-block (not in-block)) line) (and maybe-latch (setq in-block (not in-block)) nil)))))) (collect-file fileout (choose blocksp blocksp) 'write-line))) ``` # System directory stubbing I guess one should probably not want this. Still, here it is. ``` (defun stub-system (namestring) (let* ((homedir (user-homedir-pathname)) (dir (make-pathname :directory `(:relative ,namestring))) (lispdir (make-pathname :directory '(:relative "common-lisp"))) (sysdir (reduce 'merge-pathnames (list dir lispdir homedir))) (asd (make-pathname :name namestring :type "asd" :directory '(:relative :wild))) (system-asd (make-pathname :directory (pathname-directory sysdir) :name (pathname-name asd) :type (pathname-type asd)))) (ensure-directories-exist sysdir) (with-open-file (*standard-output* system-asd :direction :output :if-does-not-exist :create :if-exists :error) (format t "(defsystem ~a ~ :class :package-inferred-system)" (intern namestring :keyword)) (probe-file (make-pathname :directory (pathname-directory sysdir) :name (pathname-name asd) :type (pathname-type asd)))))) ``` # Example ``` (eval-when () (stub-system "foo") (tangle #p"/tmp/test-tangle.md" "foo") ) ``` it will error if `#p"~/common-lisp/foo/foo.asd"` already exists. It works 😺 # Conclusion While my intended focus was the first two, I found I usefully covered three topics here: 1. Tangling markdown into an `asdf` `:class :package-inferred-system` lisp system 1. Doing so with `scan-file` and `collect-file` from `series` 1. Working with lisp's `make-pathname` directories. I do think that `make-pathname` is more sophisticated than primitive string-processing style modern convention pathnames. Hope it helps someone get used to these three topics. Currently works on my machine! # Fin. I guess this article is somehow both very simple and very intricate. I would like to hear your improvements, alternatives, or memories of what-you-would-do on [the mastodon thread](https://gamerplus.org/@screwlisp/114771339388871919). Because our article shows all of - interactive, incremental, reflective functional programming with cl-series - a reasonable useage idiom of lisp's sophisticated native pathname system - working with asdf systems in a literate residential source manner as always, I would appreciate you sharing it in any particular means and manner that occurs to you personally. Small note: When I used the system on itself, my markdown file was named `#p"tangle.page.md"` so the lisp file it generated from itself used metacircularly was `#p"~/common-lisp/screwlisps-knowledge/tangle.page.lisp"` - I didn't think of how I want to deal with other programms "subtyping" file types like this, so I just fixed it directly for now. If you don't subtype files inside the filename, I guess it just works.