[HN Gopher] Show HN: Understudy - Teach a desktop agent by demon...
___________________________________________________________________
Show HN: Understudy - Teach a desktop agent by demonstrating a task
once
I built Understudy because a lot of real work still spans native
desktop apps, browser tabs, terminals, and chat tools. Most current
agents live in only one of those surfaces. Understudy is a local-
first desktop agent runtime that can operate GUI apps, browsers,
shell tools, files, and messaging in one session. The part I'm most
interested in feedback on is teach-by-demonstration: you do a task
once, the agent records screen video + semantic events, extracts
the intent rather than coordinates, and turns it into a reusable
skill. Demo video: https://www.youtube.com/watch?v=3d5cRGnlb_0 In
the demo I teach it: Google Image search -> download a photo ->
remove background in Pixelmator Pro -> export -> send via Telegram.
Then I ask it to do the same for Elon Musk. The replay isn't a
brittle macro: the published skill stores intent steps, route
options, and GUI hints only as a fallback. In this example it can
also prefer faster routes when they are available instead of
repeating every GUI step. Current state: macOS only. Layers 1-2
are working today; Layers 3-4 are partial and still early.
npm install -g @understudy-ai/understudy understudy wizard
GitHub: https://github.com/understudy-ai/understudy Happy to
answer questions about the architecture, teach-by-demonstration, or
the limits of the current implementation.
Author : bayes-song
Score : 72 points
Date : 2026-03-12 17:04 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| sukhdeepprashut wrote:
| 2026 and we still pretend to not understand how llms work huh
| abraxas wrote:
| One more tool targeting OSX only. That platform is overserved
| with desktop agents already while others are underserved,
| especially Linux.
| bayes-song wrote:
| Fair point that Linux is underserved.
|
| My own view is that the bigger long-term opportunity is
| actually Windows, simply because more desktop software and more
| professional workflows still live there. macOS-first here is
| mostly an implementation / iteration choice, not the thesis.
| renewiltord wrote:
| That's mostly because Mac OS users make tools that solve their
| problems and Linux users go online to complain that no one has
| solved their problem but that if they did they'd want it to be
| free.
| jedreckoning wrote:
| cool idea. good idea doing a demo as well.
| sethcronin wrote:
| Cool idea -- Claude Chrome extension as something like this
| implemented, but obviously it's restricted to the Chrome browser.
| rybosworld wrote:
| I have a hard time believing this is robust.
| mustafahafeez wrote:
| Nice idea
| walthamstow wrote:
| It's a really cool idea. Many desktop tasks are teachable like
| this.
|
| The look-click-look-click loop it used for sending the Telegram
| for Musk was pretty slow. How intelligent (and therefore slow)
| does a model have to be to handle this? What model was used for
| the demo video?
| 8note wrote:
| sounds a bit sketch?
|
| learning to do a thing means handling the edge cases, and you
| cant exactly do that in one pass?
|
| when ive learned manual processes its been at least 9 attempts. 3
| watching, 3 doing with an expert watching, and 3 with the expert
| checking the result
___________________________________________________________________
(page generated 2026-03-12 23:00 UTC)