[HN Gopher] Show HN: TUI-use: Let AI agents control interactive ...
___________________________________________________________________
Show HN: TUI-use: Let AI agents control interactive terminal
programs
Author : dreamsome
Score : 32 points
Date : 2026-04-08 16:41 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| wolttam wrote:
| This is kind of fun, something I've been thinking about over the
| last couple days.
|
| This is one area that makes me feel like our current LLM approach
| is just not quite general enough.
|
| Yes, developers and power users love the command-line, because it
| _is_ the most efficient way to accomplish many tasks. But it 's
| rarely (never?) our only tool. We often reach for TUIs and GUIs.
|
| It's why approaches like this get me excited:
| https://si.inc/posts/fdm1/
| mccoyb wrote:
| Something something medical researcher reinvents calculus.
|
| In 2026: frontend web developer reinvents tmux.
|
| Guys, please do us the service of pre-filtering your crack token
| dreams by investigating the tool stack which is already available
| in the terminal ... or at least give us the courtesy of
| explaining why your vibecoded Greenspun's 10th something is a
| significant leg up on what already exists, and perhaps has
| existed for many years, (and is therefore, in the training set,
| and is therefore, probably going to work _perfectly_ out of the
| box).
| dcchuck wrote:
| I agree. We skipped CLIs and went all the way to TUIs because
| TUIs are "easy to make now"? Or maybe because claude/codex?
|
| But in practice you are padding token counts of agents reading
| streams of TUIs instead of leveraging standard unix pipes that
| have been around from day 1.
|
| TLDR - your agent wants a CLI anyway.
|
| Disclaimer: still a cool project and thank you to the author
| for sharing.
| joshribakoff wrote:
| The TUI makes more sense to humans who don't understand the
| difference between a human and a machine.
| cossatot wrote:
| Maybe, just maybe, this is of obvious utility to the many
| people who have needs that are not yours?
|
| I very regularly need to interact with my work through a python
| interpreter. My work is scientific programming. So the
| variables might be arrays with millions of elements. In order
| to debug, optimize, verify, or improve in any way my work, I
| cannot rely on any other methods than interacting with the code
| as it's being run, or while everything is still in memory. So
| if I want to really leverage LLMs, especially to allow them to
| work semi-autonomously, they must be able to do the same.
|
| I'm not going to dump tens of GB of stuff to a log file or send
| it around via pipes or whatever. Why is there a nan in an array
| that is the product of many earlier steps in a code that took
| an hour to run? Why are certain data in a 200k-variable system
| of equations much harder to fit than others, and which
| equations are in tension with each other to prevent better
| convergence?
|
| Are interpreters and pdb _not_ great, previously-existing tools
| for this kind of work? Does a new tool that lets LLMs /agents
| use them actually represent some sort of hack job because
| better solutions have existed for years?
| hrimfaxi wrote:
| You can start a tmux session and tell your agent about it and
| it will happily send commands and get the output from it.
|
| I saw this post a while ago that turned me on to the idea:
| https://news.ycombinator.com/item?id=46570397
| mccoyb wrote:
| See related sibling: the use cases are compelling!
|
| My complaint is that tmux handles them perfectly. Exactly the
| claim that OP is making with their software - is served by
| robust 18 year old software.
|
| In 2026, it costs nearly nothing to thoroughly and
| autonomously investigate related software -- so yes I am
| going to be purposefully abrasive about it.
| daveguy wrote:
| And if you want to interact with tmux from within the
| python interpreter there is a very good library available,
| libtmux:
|
| https://github.com/tmux-python/libtmux
| joshribakoff wrote:
| > I'm not going to dump tens of GB of stuff to a log file
|
| In the same vein as the parent comment, the curiosity is why
| you would vibe code a solution instead of reaching for grep.
| arjie wrote:
| What I do is have a quick command that spins up a worktree on
| a repo with my ghostty splits as I like them and the tmux
| named the worktree. I then tell the Claude code about the
| tmux when it needs to look. It's pretty good at natively
| handling the tmux interactions.
|
| Ideally Ghostty would offer primitives to launch splits but
| c'est la vie. Apple automation it is.
| SatvikBeri wrote:
| Are you aware that you can use tmux (or zellij, etc.), spin
| up the interpreter in a tmux session, and then the LLM can
| interact with it perfectly normally by using send-keys? And
| that this works quite well, because LLMs are trained on it?
| You just need to tell the LLM "I have ipython open in a tmux
| session named pythonrepl"
|
| This is exactly how I do most of my data analysis work in
| Julia.
| dugidugout wrote:
| In the data science scenario you should just have proper
| tooling, for you it sounds like a REPL the agent can
| interface with. I do this with nREPL/CIDER; in Python-land a
| Jupyter kernel over MCP maybe. For stateful introspection
| where you don't control the tooling, tmux plus trivial glue
| gets you most of the way.
|
| edit: There are much better solutions for Python-land below
| it seems :)
| dcre wrote:
| Right, agents can just use tmux send-keys. Here's a skill I
| wrote to have Claude debug plugin code in the Helix editor's
| experimental plugin system. As usual, the skill is barely
| necessary, it just saves it some time getting the commands
| right and tells it where some useful reference material is.
|
| https://github.com/david-crespo/dotfiles/blob/main/claude/sk...
| reincarnate0x14 wrote:
| I sincerely think the chatbot phenomena is giving people the
| perspective that whatever hallucinatory conversation they're
| having is profound because it's the first time they personally
| have thought about it.
|
| On one hand this is normal in education and pedagogy to have
| the student or apprentice put the boring pieces together to
| find the wonder of the puzzle itself, but on the other this is
| how we end up with https://xkcd.com/927/
| nitwit005 wrote:
| The problem is, they'll find there is typically already a good
| solution to their problem, and then they'll have nothing to
| write about.
| wat10000 wrote:
| At this point, it's easier to (have the agent) build a simple
| tool like this than it is to find and set up an existing one.
| wild_egg wrote:
| I've had my agents using tmux for these use cases for a couple
| years now. What does TUI-use offer on top?
| jauntywundrkind wrote:
| I've barely been using it lately, mostly leaving it disabled.
| But the tmux-mcp is pretty solid.
| https://github.com/nickgnd/tmux-mcp
|
| I wish I was keeping better track of them all but there's a
| bunch of neat tmux based multi-agent systems. Agent of Empires
| for example has a ton of code around reading session data out
| of the various terminal uis. https://github.com/njbrake/agent-
| of-empires
|
| Ideally imo tui apps also would have accessibility APIs. The
| structured view of those APIs feels like it would be nice to
| have. And it would mean that an agent could just use
| accessibility and hit both gui and tui. For example voxcode
| recent submission does this on mac for understanding what file
| is open/line numbers. https://github.com/jensneuse/voxcode
| https://news.ycombinator.com/item?id=47688582
| petcat wrote:
| Maybe I'll use this to feed prompts into an interactive Claude
| session so I can use my max subscription instead of having to pay
| for API credits when using claude -p
| mikkupikku wrote:
| Are they any good at nethack?
| 8note wrote:
| hehe, i made something similar for feedback loop on claude hooks.
| claude can open another claude instance in the testing folder,
| and check to see if the hooks fire properly
| halfwhey wrote:
| That's neat I was working on a skill for this exact purpose:
|
| https://github.com/halfwhey/skills/tree/master/plugins/tmux
|
| Two use cases I use this for is debugging with GDB/PDB and giving
| me walkthroughs on how to use TUIs
| flux3125 wrote:
| Finally Claude Code can now control Claude Code
| alfonsodev wrote:
| I could make agents use delve (a go lang debugger) interactively,
| and it worked quite well specially when models weren't as good as
| they are now, they could choose where to put the breakpoint and
| inspect variables, I found that was the only way to unlock some
| situations when they insisted in that "it must be working", and
| it wasn't, I found that giving them the empirical tools to check
| for themselves was the only way to unstuck them.
|
| Another use was for them to read the logs out of your development
| web server ( typical npm run dev, go run .)
|
| I could do this with tmux send-keys and tmux capture-pane, you
| just need to organise the session, panes and windows and tell the
| agent where is what.
|
| That was my first agent to tool communication experience, and it
| was cool.
|
| After that I experimented with a agent to agent communication,
| and I would prompt to claude "after you finish ask @alex to
| review your code". In the CLAUDE.md file i'd explain that to talk
| to @alex you need to send the message using tmux send-keys to his
| tmux session, and to codex I'd say "when you received a review
| request from @claudia do .. such and such, and when you finish
| write her back the result of it" I added one more agent to
| coordinate a todo list, and send next tasks.
|
| After that I got a bit carried away and wrote some code to
| organise things in matrix chat rooms, (because the mobile app
| just works with your server) and I was fascinated that they seem
| to be collaborating quite well (to some extend), but it didn't
| scale.
|
| I abandoned the "project" because after all I found agents were
| getting better and better and implementing internal todo tasks,
| subagents ...etc plus some other tmux orchestrations tools
| appeared every other day.
|
| I got fatigued of some many new ai things coming up, that and the
| end, I went back to just use iTerm, split panes, and manually
| coordinate things. Tabs for projects, panes for agents, no more
| than 2 agents per project ( max 3 for a side non conflicting task
| ) I think that is also what cognitively does not tire me.
|
| My project name was cool though, tamex, as in tame tmux agents :)
|
| And to comment on the submission, I think the idea has potential,
| I might give it a try, the key is to have low friction and
| require low cognitive load from the end user. I think that's why
| skills after all are the thing that is going to stick the most.
| ofabioroma wrote:
| Does it work with any TUI?
| ofabioroma wrote:
| Does it work on any TUI?
| 6thbit wrote:
| I thought Codex at least already can handle interactive sessions
| of programs, e.g. GDB.
| kristopolous wrote:
| My version works on small local models and uses tmux under the
| hood.
|
| No installation is necessary.
|
| Simply tell your agent to run uvx agent-cli-helper and that's it.
|
| The verbiage and flow is optimized through test harnesses to
| maximize effectiveness for agentic use.
|
| https://github.com/day50-dev/acli
|
| (Sorry: my marketing and and pitch skills are trash)
| metadat wrote:
| How is acli different from telling the model to "use tmux for
| this"?
| kristopolous wrote:
| It performs substantially better.
|
| just try it out. It's not the same.
|
| There's a ONE MINUTE video showing it on the github page.
| It's just a single minute.
|
| You can also try it yourself: $ uvx agent-
| cli-helper run-command vim
|
| It sends this: <session id="vim" current-
| program="vim"> <screen-capture> 1 ~
| ~ ~ ~ ~ VIM -
| Vi IMproved ~ ~
| version 9.2.218 ~ by Bram
| Moolenaar et al. ~ Modified by
| team+vim@tracker.debian.org ~ Vim is
| open source and freely distributable ~ ~
| Sponsor Vim development! ~ type :help
| sponsor<Enter> for information ~ ~
| type :q<Enter> to exit ~
| type :help<Enter> or <F1> for on-line help ~
| type :help version9<Enter> for version info ~
| ~ ~ ~ ~ 0,0-1
| All </screen-capture> </session>
| <instructions> The command has started. To send
| keystrokes run `agent-cli-helper send-keystrokes` followed by
| the id and the keystrokes. For instance: $
| agent-cli-helper send-keystrokes vim "^X" Run
| `agent-cli-helper send-keystrokes --help` to find out the
| full syntax </instructions> <important>When you
| are done, use finish-command to finish the session. For
| example: agent-cli-helper finish-command vim</important>
| <random-usage-tip>If you need to see the current screen
| without sending keystrokes, use agent-cli-helper get-screen-
| capture <session-id></random-usage-tip>
|
| Some important parts:
|
| * it returns the output to the stdout making it easier to
| loop on
|
| * it formulates everything in fake xml so the that the agent
| knows the status and doesn't send weird keystrokes to the
| wrong command
|
| * it includes reminders and "random-usage-tip" which is not
| random, it reminds the agent of how the tool behaves. The
| "random-usage-tip" is non-random, it is called that to make
| it look unrelated but it really is.
|
| All of these things have been tested and eval'd. Calling it
| "screen-capture" "current-program", these have all been
| tested with variations for a variety of tasks/harness/model
| triplets.
|
| The tool is optimized for agent usage and has been designed
| that way as a first principle
| zmmmmm wrote:
| So if I'm understanding right, Claude Code can use Claude Code
| now?
| cyanydeez wrote:
| No, what you gonna want to do is use opencode to use claude
| code to write codex.
___________________________________________________________________
(page generated 2026-04-08 23:01 UTC)