[HN Gopher] Treefrog: A code editor that uses both AST and text ...
___________________________________________________________________
Treefrog: A code editor that uses both AST and text editing
commands
Author : gushogg-blake
Score : 90 points
Date : 2022-01-03 22:25 UTC (2 days ago)
(HTM) web link (treefrog-editor.com)
(TXT) w3m dump (treefrog-editor.com)
| gushogg-blake wrote:
| Backstory - I started experimenting with AST editing ideas after
| getting frustrated with my editor's lack of understanding of the
| structure and meaning of code, ie. the editing primitives all
| dealt with text and simple changes to the code often required
| repetitive manual text edits that had nothing to do with the
| meaning of the change. I went through a couple of prototypes and
| landed on the idea of augmenting a standard editing interface
| with a new mode that works on a simplified representation of the
| structure, which basically follows indentation levels - blocks of
| code like functions, loops, etc, can be moved and manipulated as
| units, and there are specific editing commands for the different
| elements.
|
| The main idea is to design all of the editing and navigation
| commands around the thought processes (instead of designing
| around a data structure, which I think is where traditional
| editors have gone wrong), so that it feels natural and intuitive
| to use - like something that's been designed for what editing
| code actually involves, as opposed to a text editor that's had
| lots of code intelligence stuff added on top of it.
| svilen_dobrev wrote:
| some notes:
|
| * ctrl-home/end do not go to very top/bottom
|
| * python tree-mode does not do anything sensible
| leecommamichael wrote:
| IDE-features like "Refactor" are my idea of how this kind of
| editing already exists.
| winter_blue wrote:
| A quick note: the editor looks blurry/low-res on a 4K monitor
| (and I would guess, probably on other high-res displays as well).
| gushogg-blake wrote:
| Thanks! Are you on a mac? Someone else noticed it on mac, just
| on the actual editor, so I thought it might be to do with
| rendering text on canvas (sometimes you have to use half-pixel
| coordinates to get things to align properly)
| byteware wrote:
| maybe related? https://www.jetbrains.com/mps/
| catskul2 wrote:
| The features I've always dreamed about from an AST aware editor
| was shell-like functionality where operations to rename, move,
| copy, create, etc were akin to file manipulations done on the
| command line. At one point I was thinking about the possibility
| of FUSE-
| ifying(https://en.wikipedia.org/wiki/Filesystem_in_Userspace)
| some code so that a normal shell could be used. Any chance of
| shell-like functionality from treefrog?
| gushogg-blake wrote:
| Interesting! Yes, one of the main things I'm doing at the
| moment is thinking of and prototyping new commands and ways to
| represent code. I like the shell idea.
| gushogg-blake wrote:
| So things like classes and functions would be represented as
| files within a dir representing the actual file?
| radford-neal wrote:
| I wrote a tree editor for programs for my Master's thesis, in
| 1980. You can get it at https://www.cs.toronto.edu/~radford/MSc-
| thesis.abstract.html
|
| The editor operates on the AST down to the lowest level, with
| careful design of keyboard input and cursor feedback to make that
| not too painful for things like infix operators. The language is
| described by a grammar, and there is an AST macro language.
|
| My conclusion afterwards was that the shift at that time away
| from keyboard input towards editing with a mouse was negative for
| editing in terms of the AST (rather than text), since with AST
| editing there's a less direct connection between seeing and
| pointing and what you're editing. I got interested in other
| things.
|
| But perhaps that was a premature conclusion.
| gushogg-blake wrote:
| Nice! One thing I thought ASTs weren't that good at was the
| initial writing of code - did you find that?
| radford-neal wrote:
| A big part of the design work was trying to get it to be good
| at that, while still being entirely tree based. But you'll
| have to judge yourself how successful I was.
|
| Well, your judging that would be easier if you could use it,
| rather than just read about it... I still have the source
| code, but since it was designed to work with a vector
| graphics display attached to a PDP-11, getting it to work in
| a modern environment would take some work. We had only one
| vector graphics display, so there was no chance then to have
| a significant user community, which was another problem.
| amichail wrote:
| What do you think of TeXmacs, which uses a WYSIWYG tree editor
| for writing scientific/math documents?
|
| (TeXmacs is not based on TeX nor emacs but is inspired by
| both.)
| konkin wrote:
| Math notation is hopelessly stuck in the 19th century and
| needs an update to the 20th. Polish notation with operators
| of known arity is an excellent way to ease natural
| scientists/mathematicians into a modern tree based grammar
| for mathematics without a soup of parenthesis, e.g. / 1 / 2 3
| vs / / 1 2 3
|
| A full s-expression based grammar for mathematics is
| unfortunately not something that you can write on paper and
| requires white space delineation to be readable. On a screen
| with something like emacs paredit both of those are a breeze
| however. The fact that higher order functions don't need a
| special notation is really freeing.
| Jtsummers wrote:
| What value would Polish notation add over (largely) infix
| for the typical mathematician? Why do mathematicians need
| to be told what notation to use by non-mathematicians when
| their notations (note the plural) are in wide, and
| apparently successful, use today?
|
| How would Polish notation help, for instance, with things
| like describing set membership or logical relations? How is
| _this_ more readable (using ASCII): -> a b
|
| Than the current approach: a -> b
|
| Which provides a convention sense of the relationship that
| the former does not?
| sweetsocks21 wrote:
| The S,D shortcuts work for me, but I can't get J or K to do
| anything? Some UX feedback on what is valid at the current time
| may be useful. One problem with structural/projection editors is
| it's not as clear what is a valid operation in the current state.
|
| There's some existing work in this area you might find
| interesting as well:
|
| https://twitter.com/dm_0ney/status/1414742742530498566
|
| https://hazel.org/
|
| https://hazel.org/build/dev/ (expand the left side pane by
| clicking the (?) to see the valid operations)
| gushogg-blake wrote:
| Thanks. Just checked and j/k are actually not implemented yet,
| so that explains that!
| ngvrnd wrote:
| I vaguely remember someone telling me, a long time ago, that
| Thomas Reps' PhD thesis
|
| (https://ieeexplore.ieee.org/book/6267325)
|
| showed that it wasn't practical to build this kind of editor.
| Perhaps that was not at all the point; it was a long time ago.
|
| I wonder, though, can anyone explain 1) if Reps' thesis does bear
| on the Treefrog work 2) what it was in Reps' thesis that bears on
| the difficulty of making syntax-tree based code development
| tools?
| gushogg-blake wrote:
| I wonder if it was important that an actual abstract syntax
| tree was used? I'm realising the title might be a bit
| misleading - it should probably be "AST-inspired" commands, as
| the tree-sitter trees are technically concrete as opposed to
| abstract and the commands aren't tightly coupled to a
| particular representation.
| eatonphil wrote:
| The scroll is really hard to use on my macbook. I can't scroll
| slowly enough. It moves massively up and down with each scroll
| even when I am very careful.
| gushogg-blake wrote:
| Thanks, I haven't tried it on a touchpad yet, will have a look.
| gushogg-blake wrote:
| Should be fixed now
| catskul2 wrote:
| That was fast : )
| gushogg-blake wrote:
| :) I was surprised touchpad did anything tbh - turns out it
| sends wheel events with different deltaY/deltaX values, so
| I just had to use the actual values instead of what it was
| doing before (checking if < or > 0 and scrolling by 3
| rows/columns worth)
| iddan wrote:
| This is the future of programming. Code isn't (just) text, it's a
| tree and we can edit it in a much more elegant way utilising it's
| structure
| hackinthebochs wrote:
| One pet peeve of mine that this kind of editor might be able to
| address is how live compilation is sketchy when you're actively
| editing a file. For example when you're in the middle of an edit
| and you end up with a huge number of compilation errors because
| the parse of the file is in flux. Having the editor and the
| syntax parser aware of each other can help fix such issues.
|
| Semantic merging would also be a nice feature.
| rajandatta wrote:
| Intriguing idea. This would be very interesting to try with a
| Lisp or Scheme like language. Does it have any support for S-exp
| based languages?
| bitwize wrote:
| Paredit has entered the chat.
| gushogg-blake wrote:
| I haven't added any yet, but all parsing is done with tree-
| sitter so languages can be added pretty quickly. One issue is
| that tree mode works on whole lines (I think the benefit of
| AST-style navigation vs moving a text cursor diminishes as the
| selections you want get smaller), so it's better suited to
| languages where the blocks are naturally separated by newlines
| as opposed to the ")))))" style.
| rajandatta wrote:
| Parsing on newlines is probably not a great fit for Lisp-
| syntax languages. Does the availability of LSPs make a
| difference? Asking as I don't know your tool.
| gushogg-blake wrote:
| It's more that the selection operates on whole lines in
| Tree mode, so it would parse fine with the tree-sitter
| grammar (for syntax highlighting etc) but selecting the
| inner contents of a block would take the block's closing )
| with it. This would only matter if you wanted to move the
| contents somewhere though - editing and refactoring
| commands (including with the help of LSPs) will work, with
| the selection being the outermost node on the first
| selected line.
| Jtsummers wrote:
| Yeah, s-exprs don't break down by lines very well, but they
| are easy-ish to work with in a structured format, similar
| to HTML/XML editors since the delineation between elements
| is usually very clear. The only problem with lisps would be
| when macros are involved where there is more fluidity in
| the structure. But something like this:
| (defun double-all-the-things (things) (mapcar
| (lambda (x) (* 2 x)) things))
|
| would be straightforward to navigate through with an s-expr
| aware editor.
| b3morales wrote:
| Are you using the official tree-sitter grammars as-is? Or
| tweaking/writing your own to support your use case? From what
| I've seen not all of them are suited to structural editing,
| since the emphasis is on syntax highlighting.
| gushogg-blake wrote:
| Yeah using them as is and haven't had any issues so far. I
| think they are at a slightly lower level of abstraction
| than an AST, e.g. an async method in JS is a
| method_definition with an "async" node followed by a
| property_identifier, whereas an AST might wrap all this
| into an AsyncMethod node - but all the info you need is
| there.
| buescher wrote:
| A long long time ago on a toy lisp called Interlisp/65 I used
| something that was like the s-exp equivalent of a line editor.
|
| https://atariwiki.org/wiki/Wiki.jsp?page=Lisp
|
| Source is here:
| https://atariwiki.org/wiki/Wiki.jsp?page=LispEditor
|
| If you really wanted a vi-like experience for editing lisp, you
| could figure out how to make an editor like that full-screen
| and interactive.
| billconan wrote:
| the project page doesn't anything about supported languages ?
| gushogg-blake wrote:
| Current langs are javascript, html and (s)css - there's a
| dropdown on the toolbar to create a new file in a given lang
| (the others there are partially supported, no Tree mode
| commands). The goal is to support all popular languages, as
| they can be added fairly easily with tree-sitter grammars.
| dehugger wrote:
| An explanation of what AST actually means would be helpful at the
| top of the intro page. Quite of a few of the selling points were
| lost on me due to lack of familiarity with the term. I can (and
| will) google it, but if I am looking at a new product, especially
| one I have to buy, it'd be nice to have the resources available
| to understand what it offers without turning to a third party.
| ziggus wrote:
| Yeah, I had to dredge my memory for what an AST is - ran into
| it ages ago on a compiler design project, but I could see how
| it works in a text manipulation context as well.
|
| https://en.wikipedia.org/wiki/Abstract_syntax_tree
| int_19h wrote:
| The text in the editor is blurry - looks like it's rendered at a
| lower resolution, and then upscaled with bilinear filtering.
|
| (This is on a 4K display running Win11 with UI scale set to 200%)
| gushogg-blake wrote:
| Ah yeah, it's canvas based so I suppose it will be being scaled
| like an image. Thanks!
| catskul2 wrote:
| Any chance of opensourcing this?
| gushogg-blake wrote:
| I'm not necessarily against it in the long term, just keeping
| all options open for now as I want to be able to work on it
| full time and that obviously involves making money somehow.
| catskul2 wrote:
| Understandable. I wonder if a patreon model might work.
|
| I've not hear many success stories about building companies
| around single not-yet-established pieces of software. But
| I've had similar dreams and wish you luck.
| TylerJewell wrote:
| Clever line of thinking.
|
| Is it fair to think of IDEs that include language-specific
| refactoring capabilities as a providing a developer abstraction
| over AST editing?
| gushogg-blake wrote:
| Thanks! Yeah, I think of refactorings as being a level of
| abstraction above text and AST editing - kind of like macros
| for AST edits.
| [deleted]
| luckystarr wrote:
| There has been a movement to try something like this under the
| buzzword "projectional editing". The idea was to edit a
| representation of the code and not the code itself. People and
| companies experimented with it but didn't get much adoption. Not
| sure why.
|
| Perhaps you need to allow the developers to reach invalid states,
| because the editors I tried just refused entries which would
| result in a syntax violation. This is harder to implement though.
| :)
| gushogg-blake wrote:
| Yeah, that's one of the main issues with ASTs and why I think
| they make the same mistake as text editors ultimately -
| designing around a particular data structure just constrains
| you into weird UX scenarios that don't have anything to do with
| the actual problem of making something that lets you write
| code.
| Gibbon1 wrote:
| Friend of mine said he worked with a system like that.
| Problem with it was it couldn't save code that didn't
| compile. That's a defect. Am reminded of someones comment
| that they would like a compiler that just stubs out code that
| doesn't compile. For a system to be workable you need to be
| able to handle that.
|
| I feel that because no one uses AST aware editors no one
| understands what that would bring to the table. A counter
| example I've used CAD programs. And all of them store the
| design files as databases internally. Which means you can and
| people do preform database operations on them. Generate
| reports and perform update operations just like you would
| with an SQL type database.
| sesm wrote:
| What are hypothetical benefits of using this versus
| Webstorm/Intellij with Vim mode?
| RyEgswuCsn wrote:
| Isn't this what the Intellij/Eclispe IDEs have been doing since
| the very beginning? All the powerful refactoring features are
| based on AST manipulation.
| gushogg-blake wrote:
| One example would be turning a group of statements "xyz" into
| "if (some condition) then xyz, else (something else)" - with a
| text editing interface you have to think a bit about the exact
| text selection you want (or enter visual mode in vi to get the
| lines), cut them, write the if statement, paste them in, and
| usually some reformatting, whereas with Tree mode (the
| dedicated mode for expressing meaningful edits to the code) you
| can do Esc-w for "wrap", type the if statement, and insert the
| original statements from a dedicated clipboard.
|
| Other cases are similar, e.g. being able to just drag and drop
| a div into another div without thinking about the exact text
| selection - the benefit is having to think slightly less about
| the mechanics of your editor or the particular data structure
| (text or ASTs).
| sesm wrote:
| Thanks for explanation!
|
| I think for the first case there is a 'wrap in if-else'
| refactoring in Webstorm/Intellij, but I rarely use it.
|
| Second case is interesting, I would be excited to have this
| for React components.
| NeutralForest wrote:
| Interesting, there's been an [ongoing discussion to use tree-
| sitter with Emacs](https://archive.casouri.cat/note/2021/emacs-
| tree-sitter/inde...) because right now, syntax highlighting is a
| mix of regex and functions.
|
| I don't how language parsing should be exposed to make it easy to
| have parsing in editors. But I believe some efforts like tree-
| sitter are most welcome and also more inline with what we expect
| editors to be.
| gushogg-blake wrote:
| Yeah, tree-sitter and LSP are both steps in the right direction
| I think.
| aidos wrote:
| Neovim has moved in this direction too. There's a lot of
| potential there.
|
| I hacked up a plug-in yesterday to let me run tree sitter
| queries over my codebase. I'd upgraded a dependency and some
| patterns were no longer valid so I was able to track them down
| in a way that would never have been possible with regex etc
| (certain function called with more than one argument chained
| with one of several other functions).
___________________________________________________________________
(page generated 2022-01-05 23:01 UTC)