[HN Gopher] LMDB: The first version of Redis, written in Tcl (2009)
___________________________________________________________________
LMDB: The first version of Redis, written in Tcl (2009)
Author : avinassh
Score : 155 points
Date : 2023-05-18 16:00 UTC (7 hours ago)
(HTM) web link (gist.github.com)
(TXT) w3m dump (gist.github.com)
| tpmx wrote:
| That was certainly an unusual choice of language for a problem
| like that. But, I guess, use what you know.
|
| Seems like a decent fit, I didn't expect that - I mostly know Tcl
| from very odd Tcl/Tk GUI apps in the 90s/early 00s.
| ncr100 wrote:
| It's easy to prototype string passing tools with TCL, in my
| experience with the language at Uni.
| kristopolous wrote:
| my favorite reference to it is in the HTML 4.0 standard
| https://www.w3.org/TR/1998/REC-html40-19980424/interact/scri...
|
| > "where "type" is an content type naming the scripting
| language. Examples of values include "text/tcl",
| "text/javascript", "text/vbscript"."
|
| It goes on to list text/tcl as a language, equivalent to
| javascript and even gives examples. There's a story behind this
| I read somewhere but I can't seem to find it.
|
| I once had one of those books that was like 500 pages by those
| companies like Que with a title like "The ___ Bible" which had
| at least a chapter on web tcl - in anticipation for the
| presumed support of it which I'll argue isn't around (besides
| this: https://wiki.tcl-
| lang.org/page/Using+Tcl+to+write+WWW+client...).
| lmm wrote:
| TCL/Tk was the original web-embedded language. You had
| TkLets, which were the inspiration for Java Applets and
| worked the same way (with a browser plugin etc.), but it was
| also suitable for Javascript-style scripting of webpages.
| LispSporks22 wrote:
| I have good feels about Tcl. How come it fell out of favor?
| davidw wrote:
| I have some thoughts about that, as someone who had a small
| role in that community:
|
| https://journal.dedasys.com/2010/03/30/where-tcl-and-tk-went...
| orthoxerox wrote:
| It spent too long actually storing everything as strings
| internally. When tcl 8 was finally released in 1999, tcl
| already had a reputation, people have been writing cgi-bin
| scripts in Perl for 6 years already and PHP was on the rise.
|
| Homoiconicity is cool, but cool doesn't necessarily mean
| useful.
| sokoloff wrote:
| I've written a lot of Tcl in my career. I don't know if it
| "fell out of favor" as much as "never really had a ton of favor
| even at its peak". I have used it in the last decade to do some
| build system tooling where string/regex processing of log files
| emitted in prior steps was a big part of the functionality. It
| was nice that "nobody else is using Tcl, so I won't have any
| versioning conflicts" and that held true for over a decade, so
| the long-term cost was reasonable.
|
| If I had to learn a new language today, I can't say that I'd
| put Tcl very high on the list, despite being overall happy with
| it.
| themerone wrote:
| I read an opinion that the TCL community was divided into two
| factions. One faction wanted a lua like small embeddable
| interpreter. The other wanted a full feature Python like
| platform. Neither faction won, so TCL got stuck in the middle.
|
| I don't know if this is true, but it was such an interesting
| opinion, that it stuck with me.
| IshKebab wrote:
| We got much better options... Basically anything really but
| even not-very-good languages like JavaScript and Python are
| _significantly_ better than TCL. The only thing that TCL really
| beats is Bash.
|
| If you are restricted to easily embeddable languages then the
| options are much more limited, but even so there are obviously
| better options like Lua, or even rolling your own (e.g.
| GDScript, QuakeC).
|
| The only people still using TCL are those that can't leave 90%
| in the past, e.g. the entire EDA industry.
| Cyph0n wrote:
| Still huge in the circuit design/EDA (electronic design
| automation) space.
| Aromasin wrote:
| Yup, used in all the FPGA software tools. A lot of the new
| stuff coming out of Quartus at least is being written in
| Python though, or rewritten as such. Having worked in the
| field for the past year, I've definitely got quite a few
| gripes with the Tcl after some exposure.
|
| The syntax is not intuitive, especially for a scripting
| language. The standard library is far too small, basically
| just a subset of the C standard library plus regexps. The
| only true data types in Tcl are strings and associative
| arrays. As such, Tcl scripts do not scale very well. The
| performance is generally abysmal, not just because of the
| magic string conversions, but because standard library
| functions are incredibly slow in cases that are hard to
| avoid. Tcl fully parses lines before deciding that they
| should be executed (in the case of a command) or ignored (in
| the case of a comment). You should think of a comment as a
| "do nothing" command, rather than as a comment as in other
| languages. While not too egregious, if you're used to other
| languages that quirk causes numerous grievances.
|
| One positive is it's really easy to drop down into C or C++
| when you come across tasks that make more sense in a lower-
| level language, so if you're using it for that purpose it's
| good.
| bch wrote:
| > Tcl fully parses lines before deciding that they should
| be executed (in the case of a command) or ignored (in the
| case of a comment).
|
| I don't know your measurements or use case but putting code
| in a [proc] will engage the bytecode compiler which might
| help you out?
| Gordonjcp wrote:
| VHS vs. Beta, basically.
|
| Tcl and Perl are about the same age, approximately similar in
| function, approximately similar in ease-of-use, and really the
| world could have gone either way.
|
| Just as with VHS and Beta, each had their strengths and
| weaknesses, but ultimately it came down to an avalanche of the
| "network effect" - everyone was using VHS so everyone used VHS
| / everyone was using Perl so everyone used Perl.
| VWWHFSfQ wrote:
| It has some niche use-cases but I think in general it was just
| too awkward. "Everything is a string" kinda puts people off.
| Lua has similar weirdness with various aspects but it's still
| highly successful in other niche areas.
|
| I happen to really like Lua. But a lot of people don't like the
| 1-based indexing and table container can be difficult for
| people get wrap their head around.
| jollyllama wrote:
| Yes, it makes debugging a large or distributed TCL
| application hellish. Once you have had to deal with that, you
| will be put off of TCL for life.
| canadianfella wrote:
| [dead]
| bch wrote:
| > "Everything is a string" kinda puts people off.
|
| I can't speak for "people", but i think this might be a bit
| of misunderstanding combined w an implementation detail.
|
| I think most Tcl people these days would say "everything is
| representable as a string", which is to say types can be
| marshalled back/forth through a string representation. This
| (with other features) allows Tcl to be homoiconic[0] (it can
| evaluated the symbols it emits itself). Pre-8[1]
| (1997/1998(?)) Tcl internals did also indeed use char*
| strings to do its own work parsing/evaluating, but Tcl8
| brought in so called Tcl_Obj structures w "dual ported"
| values that keep the iconic string representation available,
| as well as a native value (ie: integer when using the value
| as an integer, so you wouldnt need to do something like
| atoi() on a string rep when you wanted to do math).
|
| [0] https://en.wikipedia.org/wiki/Homoiconicity
|
| [1] https://tcl.tk/software/tcltk/8.0.html
| IshKebab wrote:
| Semantically everything is a string, even if they have
| performance optimisations to make that slightly less
| insane.
|
| Performance is only one of the issues with "everything is a
| string". Bug-proneness is probably a bigger issue and you
| can't fix that with optimisations.
| lockhouse wrote:
| I think there were several factors.
|
| While Tk (one of Tcl's major killer apps) looked good on
| Windows and Mac, it retained the Motif look and feel for too
| long on UNIX platforms. This gave the impression of it being
| obsolete technology and made many people dismiss it for use in
| serious applications.
|
| It lacked a major company championing and investing in it.
|
| Its creator Dr. Ousterhout, moved onto other projects.
|
| Python, JavaScript, Ruby, and PHP won out on the scripting
| language popularity contest at the expense of Perl and Tcl.
|
| Richard Stallman had very strong opinions about Tcl that also
| did it no favors. This was known as the "Tcl War."
|
| https://vanderburg.org/old_pages/Tcl/war/
| IshKebab wrote:
| > If Tcl does become the "standard scripting language", users
| will curse it for years--the way people curse Fortran, MSDOS,
| Unix shell syntax, and other de facto standards they feel
| stuck with.
|
| Hear hear. Just Bash left to kill (and TCL in EDA but I won't
| hold my breath there - hardware people think they aren't
| software engineers so they don't care about software
| quality).
| zerr wrote:
| So the Lisp-like is Guile. Did they ever come up with the
| second (algebraic) language?
| ncr100 wrote:
| I am tickled to see this, first Redis implementation in TCL.
|
| TCL was (is?) slow and there was not the userbase that needed
| it, at the time, in my view.
|
| Maybe now with more diverse programmers addressing more diverse
| use-cases with their software, TCL can be more relevant.
| nobleach wrote:
| I still recall coming across the "Sams Teach Yourself Tcl/Tk in
| 24 Hours" book at my office. I had been doing Linux Samba
| administration at that time and some of the tooling was written
| in tcl. So I brought the book home. It was the first time I had
| ever doing any GUI stuff that wasn't Delphi/VB/FoxPro! I loved
| it! I quickly realized I had nothing interesting I needed to
| build at that moment... so I put it back down. But just
| realizing that GUI programming was in reach without all the
| heavy Win32/GDI or Winforms, was so cool.
| zerr wrote:
| I felt the same, but with GTK.
| ilrwbwrkhv wrote:
| It's amazing seeing the init version of arguably one of the most
| popular software of the modern era.
|
| Goes to show how much programming is like writing. You keep
| iterating and making it better.
| didip wrote:
| I think this is relevant... These are 3 OSS databases that can be
| an alternative to Redis:
|
| - KeyDB: https://github.com/snapchat/keydb
|
| - Dragonfly: https://github.com/dragonflydb/dragonfly
|
| - Skytable: https://github.com/skytable/skytable
|
| I have used keyDB before. The raft consensus makes building an HA
| Redis easy.
| mobilio wrote:
| Also SSDB: https://github.com/ideawu/ssdb
| danogentili wrote:
| We use keydb at work, and I absolutely do NOT recommend it due
| to its extreme instability, in fact we're currently in the
| process of switching to dragonfly precisely due to keydb's
| instability.
|
| Multimaster replication is convenient, but if you accidentally
| add the IP of the current replica to the replicaset, the entire
| cluster will crash if any of the nodes shuts down for any
| reason (note, not immediately, but days or even weeks after
| starting the cluster with everything working perfectly until
| one of the nodes crashes or shuts down).
|
| Enabling diskless replication in multimaster mode causes either
| deadlocks, or immediate crashes, depending on the overall LA of
| the system (!).
|
| These are just the issues I remember right now, because the
| related issues I opened in the repo are still open almost a
| year later; we had other crashes we had to workaround, too.
| jdsully wrote:
| Sorry it didn't work well for you. In the 6.3 release we
| tried to pack in too much although we've ironed out most of
| the issues. The hardest change has been moving away from
| fork() for background saving which has historically caused a
| lot of pain for running it in production and makes it
| difficult to do persistence.
|
| I definitely would have paced things out better if I could go
| back in time.
| antirez wrote:
| Yep in general when people consider Redis forks or
| alternaives that use more complex models, they don't realize
| the cost associated to the fact Redis is made in a certian
| way to be very stable and solid, and different choices lead
| to more fragility. Things that sometimes gets unnoticed:
|
| 1. The non threaded architecture of Redis need more
| management since you have to shard to many instances, but _it
| is optimal_ from the POV of CPU utilization, especially if
| you do pinning to different NICs and so forth. And the risk
| of bugs in threaded code is very large.
|
| 2. Redis threadoffs such as copy-on-write during persistence
| is not optimal all the times, but the worst case behavior is
| predictable, and whatever is the layout of the data
| structure, you copy 4k pages. When more complex techniques
| are emplyed, sometimes you can do much better, sometimes you
| fail in a quite pathological way.
|
| 3. In general simplicity is there for a reason, and if you go
| with a fork that is more compelx, has ways less users, less
| developers and so forth, things can be less pleasent than
| they appear in READMEs with fancy benchmarks.
|
| 4. Redis is conceived to be easy to _understand and modify_
| by the folks that run it at scale. So the simplicity also is
| a way to say: come in, change things in your fork, make it
| particularly adapt for your use case: it 's not going to be
| too complex to do it. Hackability is the contrary of vendor
| lock-in.
| avinassh wrote:
| Dragonfly is not open source, rather source available.
|
| Also, has anyone here used SkyTable as an alternative to Redis?
| wizzwizz4 wrote:
| Dragonfly's source available license reverts to Apache on
| 2028-03-15. I think this "Business Source License" is a
| _wonderful_ idea: there are business reasons to have
| proprietary code (yet another indictment of how we 've
| structured our society), but this retains a decent chunk of
| the long-term benefit-to-the-user that open source provides
| (namely, reduced vendor lock-in; ability to exercise GDPR
| data portability rights; ability to keep stuff going after
| the company goes bankrupt and its assets get bought by a
| creative-tax-accounting "business").
|
| "Production" doesn't seem well-defined, so in practice this
| is probably a crayon license; and reverting to Apache gives
| you all the problems of permissive licenses. But it's a
| wonderful _idea_!
| tuppinsabound wrote:
| Also Opatomic https://opatomic.com/
|
| Database server is not yet open source though
| hivacruz wrote:
| We use Kvrocks[0] at work. It is Redis on disk, powered by
| "RocksDB" (hence the name) and compatible with most of the
| Redis clients since it respects the Redis protocol. It was
| incubated by Apache earlier this year.
|
| It works great and the development is really active.
|
| [0]: https://kvrocks.apache.org
| torh wrote:
| I would recommend going back 9 days and read the story behind
| Redis: https://news.ycombinator.com/item?id=35871462
| dang wrote:
| Thanks! Also related:
|
| _LMDB - First version of Redis written in Tcl_ -
| https://news.ycombinator.com/item?id=14230960 - April 2017 (25
| comments)
|
| _Redis_ - https://news.ycombinator.com/item?id=494649 - Feb
| 2009 (11 comments)
| ben7799 wrote:
| I'm more surprised Tcl was used in 2009 than I am at any praise
| or complaint about the quality of Tcl.
|
| I used Tcl and Expect (which was a very cool extension of Tcl) in
| the late 90s in some automation projects as an intern. It did the
| job and yet I was aware of some of it's issue.
|
| But that was so far before 2009. It felt like Tcl was almost done
| by 2000, I never saw it used again. Perl came and went, Lua had
| arrived, Python had arrived, etc..
|
| Tcl to me is like Python.. pretty good for small projects by a
| small team. Not so desirable for a giant enterprise
| product/platform with a big team.
| dfinninger wrote:
| I _started_ writing TCL in 2010. This was for EAD automation,
| and the software only had a TCL API, so probably just a
| historical quirk. But the company I was with had a lot of it to
| automate a lot of flows.
| antirez wrote:
| I started with the goal to write a C implementation just after
| I could verify the idea. Yet this is a list of reasons why,
| back then at least, Tcl was a great choice to write a
| prototype:
|
| - Tcl reference counting automatic mem management is extermely
| solid: it does not introduce any pause and is very consistent
| in long running processes. Another language I love is Ruby:
| when I used it for similar reasons, a few years later, I was
| shocked by how impossible it was to run long running processes
| in Ruby because of the garbage collector limitations.
|
| - Now many languages support event-driven programming very
| well, but Tcl used to have this for many many years in 2009, it
| was rock solid.
|
| So Tcl allowed to write something that was not fragile, but
| only limited in speed and with a large memory footprint. And
| that served very well as a proof of concept to validate the
| idea, and then switch to the C implementation.
|
| This code never ran in production, if I remember correctly: I
| stopped the experiment as soon as I saw it working well _and_
| as soon as I understood the fork-based persistence I wanted to
| implement was impractical in Tcl.
| klodolph wrote:
| Tcl is super interesting from a historical point of view. It
| seems to hit about halfway between shell and Lisp.
|
| With some small differences, it probably would have competed
| more directly against Perl, Python, or maybe Lua. As it is, it
| gave us Wish, plus a bunch of GUI wrappers for shell scripts,
| and we're thankful for those.
| ben7799 wrote:
| I think some of it back then was there was less internet
| communication and Tcl could be popular one place and Python
| was popular in another place and Perl popular in another
| team.
|
| Now we discuss everything so much communities converge on a
| common choice.
|
| Perl & Tcl really share the issue that undisciplined
| developers could make really obfuscated code that was hard to
| decipher.
| [deleted]
| sgt wrote:
| Makes me wonder, are there any current projects still using
| tcl? I know DDG is still using perl.
| formerly_proven wrote:
| Python ships with Tcl/Tk for tkinter.
| bch wrote:
| F5 iRules, Cisco iOS (afaik), embedded in network storage
| devices, Argonne National Laboratory Swift/T supercomputer
| control, ...
| cmacleod4 wrote:
| Various projects and companies using Tcl are listed at
| https://wiki.tcl-lang.org/page/Who+Uses+Tcl .
| sgt wrote:
| If it just had a fresher UI it would be just fine for most
| users.
| cozzyd wrote:
| tcl is heavily used in electronic design tools (e.g. Vivado).
| fweimer wrote:
| gitk and "git gui" are written in Tcl/Tk.
|
| Some components of the GNU toolchain use dejagnu in their
| testsuite, which is based on expect and written in Tcl.
|
| The SQLite testsuite uses Tcl, too.
| Klonoar wrote:
| MacPorts.
| lmm wrote:
| TCL had great async I/O back when virtually nothing else did.
| In a time that was pre-nodejs (I just checked, started in 2009)
| and when Python async I/O was still a mess (Twisted worked
| great, but it wasn't easy to work with), TCL would've been the
| obvious choice for a high-performance network server. (See e.g.
| AOLServer).
|
| (I mean, the functional world had proper async support at that
| point - I'd probably have used Scala or OCaml - but a lot of
| people are scared of those languages)
| yamtaddle wrote:
| > I used Tcl and Expect (which was a very cool extension of
| Tcl)
|
| I used "expect" about five years ago to help automate deploying
| an app to a local Roku box from the command line. Much faster
| dev loop than having to upload a zip to a webpage hosted on the
| Roku box.
| gilbetron wrote:
| My first job out of college used TCL in 1992, and I made a
| decent sized application with it. It was neat, but ultimately
| weird and kind of unwieldy, and definitely fraught with peril!
| In 1995, I left the job behind, thinking I'd never see TCL
| again.
|
| Fast-forward to 2011 and I found myself in a new job where I
| was pointed at their middleware to do some enhancement, only to
| find myself in a TCL application yet again! Hopefully that was
| the last I'll have to deal with it professionally, although it
| is kind of a nifty language.
| gt565k wrote:
| mIRC tcl scripts and tcl scripts for Eggdrop IRC bots
|
| oh man, the nostalgia!
| [deleted]
___________________________________________________________________
(page generated 2023-05-18 23:01 UTC)