[HN Gopher] Why Turtl switched from Common Lisp to JavaScript (2...
___________________________________________________________________
Why Turtl switched from Common Lisp to JavaScript (2019)
Author : e3bc54b2
Score : 59 points
Date : 2021-10-27 18:05 UTC (4 hours ago)
(HTM) web link (lisp-journey.gitlab.io)
(TXT) w3m dump (lisp-journey.gitlab.io)
| vindarel wrote:
| On this:
|
| > I don't have to deal with quicklisp's "all or nothing"
| packaging that doesn't support version pinning.
|
| it's easy to clone a given version on quicklisp's local-projects,
| but we also have Qlot and the emerging CLPM. See also the new
| https://github.com/tdrhq/quick-patch/
| orthecreedence wrote:
| I made great use of local-projects when Turtl's server was
| written in CL, and at the time I'm not sure if Qlot/CLPM
| existed (never heard of them until your comment). This was many
| years ago the switch happened.
| tdeck wrote:
| The first link is to https://turtlapp/ which is broken; I think
| it should be https://turtlapp.com/
| vindarel wrote:
| Oh my. Thanks, should be fixed in a minute.
| snicker7 wrote:
| > however when things did go wrong, there was nobody to run ideas
| by and nobody to really help me...I had built all these things
| myself, and I also had to be responsible for fixing them when
| they broke.
|
| You are responsible for your supply chain. I've had instances at
| work where some rando open source library would be bugged, and
| we'd have to patch it up ourselves. It is much easier to fix code
| you've wrote.
| draw_down wrote:
| Given enough eyeballs, all bugs are shallow ;)
| orthecreedence wrote:
| > You are responsible for your supply chain.
|
| Exactly, which is why I abandoned my supply chain! I bit off
| more than I could chew with CL/async.
| I_am_tiberius wrote:
| I thought they switched from Lisp to Rust. Was that process
| cancelled?
| orthecreedence wrote:
| The backend switched from CL -> Nodejs, the frontend switched
| from JS -> Rust core bundled with JS ui.
|
| There was an experimental app core written in CL
| (https://github.com/turtl/core-cl) that is basically the
| precursor to what the rust core is today. I did have success
| embedding lisp (via ECL) into other components (like node-
| webkit at the time) but ultimately it was kind of an uphill
| battle, as with many off-the-beaten-path things in common lisp.
| gibsonf1 wrote:
| Strange, there is a great web server woo [1] and great parallel
| futures library for async lparalel [2] that we use with great
| results.
|
| [1] https://github.com/fukamachi/woo (4x faster than nodejs) [2]
| https://lparallel.org/
| orthecreedence wrote:
| _As far as I know_ , Woo has no async foundation, meaning it
| can serve static content very quickly, but cannot really make
| use of other async libs without farming out to another event
| loop/thread pool.
|
| In other words, its main asset, speed, is bottlenecked as soon
| as you need to _do things_.
|
| Wookie, the original server for Turtl, was built on top of cl-
| async which gave it access to all the other async libs built
| for CL. Woo wouldn't have been particularly useful for us.
| gibsonf1 wrote:
| That would be a misunderstanding of Woo. You can have as many
| workers running as you like with woo, but for async
| activities processing requests, the issue isn't woo, that is
| the server doing things with the request once received, and
| for that you run lparallel with great results. You can simply
| send every request off in a future, and it will go off and do
| it on another thread.
| orthecreedence wrote:
| So you're not running anything truly async, you're just
| using threading at a lower level? I don't get what Woo
| really provides to the async ecosystem in CL.
| gibsonf1 wrote:
| Woo is a superfast web server. It handles receiving 4x
| the number of requests that node js can. Once the server
| gets the request, you can control how the response works
| with futures etc. Our front end webapp is using Vue cli,
| which clearly is javascript. I assumed the discussion
| here was about the server, not the front end webapp on
| the browser.
| wyager wrote:
| I wouldn't describe 4x nodejs as "super fast". Nodejs is
| one of the slowest popular backends, or at least it was
| when I did a comparison a few years ago.
| gibsonf1 wrote:
| In the last benchmarks, woo beat all including the Go
| webserver: Scroll down just a bit to see the comparisons
| here: https://github.com/fukamachi/woo
| orthecreedence wrote:
| The discussion was about an app server written in CL
| (Wookie) that supports a backend app written in CL, all
| in async (via cl-async).
|
| It seems to me you're using Woo as an Nginx replacement,
| not as a CL app server, which makes total sense to me
| after you described the model you're using.
| Jtsummers wrote:
| What is your distinction between "async" and "using
| threading at a lower level"?
| orthecreedence wrote:
| In this case, an async (via cl-async) application served
| by an async app server (http://wookie.lyonbros.com/), so
| async all the way down using evented I/O vs an async
| server (Woo) that farms out all requests to a synchronous
| thread pool.
|
| I've been asked multiple times why I "didn't just use
| Woo" by people who don't understand that Turtl's server
| was _async_ and Woo _doesn 't support_ async.
| throw10920 wrote:
| tl;dr three major pain points:
|
| (1) no native async
|
| (2) small library ecosystem
|
| (3) CL community NIH syndrome:
|
| > I think the straw that broke the camel's back was when a few
| people started making copycat projects that added no real value
| (other than benchmarking fast) but stole mindshare from all the
| work I had put in.
|
| These are good points. (1) is a real, true problem with CL (as
| opposed to things that people commonly think of that aren't
| actually that bad e.g. syntax), and it's probably not going to be
| solved in a satisfactory way anytime soon - the design of the
| language is pretty nice, but seems to exclude first-class
| continuations (according to [1]), async, and coroutines. (yes,
| you can _emulate_ these things using macros, but that often
| involves tree-walking, and results in non-orthogonal features
| that don 't play great with the rest of the language - first-
| class is always better)
|
| Common Lisp was designed for, and is very good at, representing
| _algorithms_ under a very specific model of how the world works -
| that is, single-node and single-memory-space. Its design
| assumptions don 't hold up very well in a multi-node, concurrent
| world. Yes, there's bordeaux-threads and lparallel, but they feel
| like hacks, and the presence of GC means that they won't be able
| to attain the performance of Rust or Erlang (which seems like
| _Lisp for a distributed world_ - its model of the world seems to
| be _fundamentally_ distributed, not a bunch of patches on top).
|
| (2) is somewhat unavoidable when you have a small community, but
| (3) is not (and is _highly_ undesirable). Ideally, the CL
| community would document all of their published code really well
| (from the library-author side), and then put some effort into
| learning how to use (and improve) someone else 's code (from the
| library-user side) instead of rewriting their own 50%-complete
| version of an existing library, but that would require a
| significant shift in the attitude of the average CL user, and I'm
| not sure how to try to affect that.
|
| [1] https://stackoverflow.com/questions/16651843/why-doesnt-a-
| pr...
| vindarel wrote:
| I think (2) should be more "small ecosystem of _ASYNC_
| libraries ", that's what he complains about.
| [deleted]
| math-dev wrote:
| Yeah the CL community likes to be lone warriors, I am
| definitely very guilty of that.
|
| Documentation in CL is pretty bad. Worst is when people say its
| well documented and its nothing but, all those manuals just
| listing all the functions are a complete nightmare and useless
| to me
| throw10920 wrote:
| I'm guilty of this too! Change begins with one's self, they
| say, heh. (I should learn to write good docs before
| castigating the community)
|
| And you're right - trivial docstrings for all the functions
| isn't documentation at all. Real documentation is the four-
| part system[1] of tutorials, howtos, examples, and reference
| material - I've found this mental framework to be enormously
| helpful for both (a) identifying when some other
| documentation is inadequate and (b) improving my own.
|
| [1] https://documentation.divio.com/
| jes wrote:
| I'm interested in other people's thoughts on this.
|
| In the Lisp world we often talk about "exploratory programming."
| There are contexts, or at least there have been in my career,
| when I sit down to work on a particular problem and I don't know
| ahead of time how it's gonna work out or how I'm going to solve
| it. In these contexts I find Common Lisp to be a useful tool.
|
| In other contexts I have a clear understanding of what I need to
| do, and so writing software in C++ or some other language is
| fine.
|
| I'm not really interested in trying to shoehorn one solution into
| another context. I'd say use the tool that works best for you in
| the context in which you find yourself.
|
| I could be wrong. Interested in your thoughts if you want to
| share them.
| klodolph wrote:
| Exploratory programming, for me, involves tons of quick
| refactoring jobs. Moving methods around, copying code from one
| place to another. If I sit down with a decent Go, C#, or Java
| IDE, I can sling code left and right, mash it up, refactor it,
| move code around, change interfaces, etc.
|
| The good tooling for these languages makes it so I can do a lot
| of this work with a couple button presses. I inevitably end up
| with code made from different phases of the exploration... the
| static type checker is good at reminding me which pieces of the
| code base need to be updated, because they were from an earlier
| phase of exploration. There are a lot of idiosyncratic tricks
| buried in here, like changing the name of a method when I
| change its semantics (like frobnicateWidgets() to
| frobnicateWidgets2()), so all the call sites become errors, and
| I can review them one by one.
|
| This works poorly in C++, the tooling just sucks.
| brobdingnagians wrote:
| I agree that tooling for JVM languages, Go, and C# are better
| at that, but clang and Jetbrains' CLion has helped close that
| gap. They've done a lot to make refactoring a lot easier,
| find errors better, and not have to compile just to find
| obvious mistakes. Newer languages are much better designed
| for IDE/tooling support in general, with stronger type
| systems in a lot of cases.
| klodolph wrote:
| Jetbrains has put a similar amount of work into tools like
| Rider, it's just that they're starting from a language
| where these problems are much simpler to deal with.
| japhib wrote:
| Is there something special about CL as compared to other Lisp
| dialects? Why not choose something like Clojure, or LFE (Lisp-
| Flavored Erlang) with better async/library support? Then you skip
| over the community aspect entirely by piggybacking off the JVM or
| Erlang communities.
|
| (Another alternative would be Elixir, which is basically Erlang
| wrapped in a nicely-formatted version of Lisp. I.e. Elixir is to
| Erlang what Clojure is to Java.)
|
| I'm sure the answer comes down to something like, the author was
| more familiar with Node.JS so it would be a very quick rewrite.
| But I feel like there are a lot more Lisp-like language choices
| for backend APIs before you jump to Javascript.
| cellularmitosis wrote:
| The article describes making use of one of the key features of
| Common Lisp implementations, that you can change both data and
| code on the fly while in the middle of an exception. I'm not
| certain those other lisps allow for that level of dynamism.
| medo-bear wrote:
| > Is there something special about CL as compared to other Lisp
| dialects?
|
| cl is ansi standardized. it has several fast industrial
| strength implementations available that are self-hosted. it is
| both a high and low level language. it is FAST ... way faster
| than the dialects you named. its interactive development env is
| second to none
|
| > Elixir is to Erlang what Clojure is to Java
|
| i think LFE is meant to be that for Erlang. and i think that
| calling Elixir a lisp is stretching definitions quite a bit. as
| an asside there is an implementation of common lisp that
| resides on JVM - https://www.abcl.org/
| patrec wrote:
| > Is there something special about CL as compared to other Lisp
| dialects?
|
| It has by far the best interactive development experience.
| epolanski wrote:
| Why? Asking as someone who only knows some Scheme.
| [deleted]
| vindarel wrote:
| Hey guys, don't miss this nice interview with Kina Knowledge:
| https://lisp-journey.gitlab.io/blog/lisp-interview-kina/
| (https://news.ycombinator.com/item?id=28961987)
| bestcoder69 wrote:
| This is the crux of the issue but it's not explained further:
|
| > I wanted to be able to use asynchronous programming because for
| the type of work I do (a lot of making APIs that talk to other
| services with very little CPU work) it's hard to get much more
| performant.
|
| Presumably the author of Turtl hit (or predicted) a scaling
| bottleneck with synchronous CL? Does CL not perform very well
| with tons of threads doing synchronous i/o (compared to other
| languages)?
|
| I also don't see a lot of upsides here for CL. The pros given
| are:
|
| * You can modify code while it's running so it's useful for
| debugging. Is this so much more useful to warrant switching
| languages from a language where I just re-run my whole unit test
| after modifying the implementation? They tend to be imperceptibly
| quick.
|
| * It's elegant, fast, and stable. Compared to NodeJS I guess I'm
| willing to just accept these :P
| Jtsummers wrote:
| CL doesn't have a truly standard way of writing async/multi-
| threaded code. Every (practical) implementation supports native
| OS threads, and Bordeaux threads supports every reasonable
| implementation while providing a uniform way to access their
| lower level (and implementation specific) versions. You _can_
| get good async /multithread performance out of Common Lisp. But
| it's, by definition, non-standard. Node at least made asyncio a
| part of its standard implementation, so you're not having to
| pick and choose or construct it from lower level primitives.
| There are other higher level concurrency libraries for CL that
| build on BT and the lower level primitives to remove the need
| for you to think about it, but they're still non-standard.
|
| That's all missing from CL as a part of the standard, if CL
| were to get a second standard something like that would almost
| certainly make it into the revision. But it's not getting
| another standard any time soon.
| bestcoder69 wrote:
| > if CL were to get a second standard
|
| Huh, TIL. Checking wiki, the final/current version is from
| 1994[0]. Welp, that explains it for me. Thanks!
|
| 0: https://en.wikipedia.org/wiki/Common_Lisp#History
| gibsonf1 wrote:
| We simply use lparallel with incredible results. We combine
| that with using a thread-safe version of cl-containers, such
| that we save futures in a container, and then when needed,
| force the futures in the entire container.
|
| https://lparallel.org/
|
| https://github.com/gwkkwg/cl-containers
| gknoy wrote:
| > With CL, it felt like I was constantly fighting the tide....
| just to get basic things working that are already solved
| problems in other languages (Node).... I was mostly fighting it
| alone.
|
| Based on the author's comments in the article, I didn't get the
| impression that he hit any performance bottlenecks -- rather,
| it was an emotionally draining experience to be on the hook for
| everything. Similar to how some open source authors walk away
| from things because it's "Too Much".
| bestcoder69 wrote:
| Yep that part makes sense to me. But I meant: if there's no
| perf bottleneck why care about async i/o at all? Since that's
| the impetus for async i/o usually. A sibling comment to yours
| answered that part for me. My understanding now is that
| concurrent code in any paradigm is going off the beaten path,
| i.e. away from standardized CL, so at the very least you
| aren't going to be able to write concurrent code portable
| across different CLs.
| Jtsummers wrote:
| Async IO is not chosen because a language isn't fast
| enough, it's because _IO_ isn 't fast enough. Async IO
| would be useful even if you handwrite everything in the
| best and fastest assembler possible because in the time it
| takes to go out and read some data (even on an SSD) you can
| compute many, many more things. The purpose of async IO is
| to permit the interleaving of fast and slow activities. If
| I were still doing scientific computing (it's been a while)
| I'd want async IO (some implementation of it) even if I had
| the best GPU for my number crunching bits. Loading the
| model takes a lot of time and there are other activities
| the program can engage in during that delay.
|
| CL's lack of a standard way to do async IO (in this
| particular instance) means that even if you have a very
| fast program, you end up with either a bottleneck waiting
| for IO or you've recreated the async IO model from other
| languages (or their standard libraries or their de facto
| standard libraries) and are now responsible for it.
| brazzy wrote:
| > The purpose of async IO is to permit the interleaving
| of fast and slow activities.
|
| You absolutely do not need async IO for that. You can do
| that with a synchronous programming model just fine,
| using threads and letting the OS do its thing.
|
| Async is an _absolutely horrible_ programming model.
| There are really only two advantages is brings: It lets
| you avoid the memory and context switching overhead of
| threads if you need to handle a very large number of
| requests concurrently, and it lets you do concurrency
| without worrying about synchronizing access to shared
| resources (but there are much better ways to do that).
| Jtsummers wrote:
| If you use threads to cordon off the slow IO code, you're
| doing async IO, just without core language or library
| support for it.
| orthecreedence wrote:
| Yeah, admittedly I could have gone with
| threading/hunchentoot and been much better off. It was
| probably stupid to forge the entire async path the way I
| did.
|
| To be fair, Turtl _these days_ gets enough traffic where
| async does probably make sense for us, and nodejs is
| serving us well.
___________________________________________________________________
(page generated 2021-10-27 23:00 UTC)