[HN Gopher] Show HN: Simplex: Automate browser workflows using c...
___________________________________________________________________
Show HN: Simplex: Automate browser workflows using code and natural
language
Simplex is an SDK that provides low-level functions you can use to
create reliable web automations using code and natural language.
Here's a quick video to show you how it works:
https://www.loom.com/share/8d27d0f9e0f34c77a81b0ac770151c12 A
couple weeks ago, we needed a way to automatically find 3D assets
from the internet for one of our work contracts. We didn't need an
AI to choose all the steps for us autonomously -- we knew generally
what the flow to find items should be and just needed some
intelligence to handle different webpage formats. Playwright
couldn't easily generalize to the different sites we wanted to
search over and Claude computer use was tough to use -- it was
slow, expensive, struggled to click on the correct web elements,
and there was no way to correct its actions if it failed at step 20
of our workflow. So we built a vision-based solution using vision
language models that sits on top of Playwright. In our SDK,
building a function that can universally search websites is as easy
as this: from simplex import Simplex
simplex = Simplex(api_key="api_key") def
search(website, search_query): simplex.goto(website)
simplex.click("the search bar")
simplex.type(search_query) simplex.press_enter()
search("mit.edu", "alyssa p hacker") You can play around with
what we've built and see a few more examples at
https://simplex.sh/playground. We'd love feedback!
Author : marcon680
Score : 129 points
Date : 2025-01-14 21:30 UTC (1 days ago)
(HTM) web link (www.simplex.sh)
(TXT) w3m dump (www.simplex.sh)
| xnx wrote:
| Looks pretty cool. How do you distinguish Simplex from Skyvern or
| UI.Vision?
| marcon680 wrote:
| Hey thanks! We think we're pretty different from Skyvern as
| Skyvern provides a full agent loop for users + is no code. We
| wanted to build something at a lower level (i.e., no high-level
| planner that chooses what tasks to do for you) and we wanted to
| be able to program our own logic alongside the intelligence
| part because using code is what's natural for us as developers.
|
| I hadn't heard of UI Vision but just took a look at it -- it
| also looks like a no-code solution that's a Chrome extension,
| so I'd say the main differences are the same as the differences
| w/ Skyvern -- we're lower level and meant to be used by
| developers.
|
| I'd add that we're also able to directly extract parts of
| websites that have no official API -- for example, an image of
| the GitHub contribution graph like I show in the video demo.
| picografix wrote:
| it fails for this query search("amazon.in", "fitness watch")
| marcon680 wrote:
| Ah, we've whitelisted some websites to prevent abuse --
| amazon.in wasn't one of them. I just whitelisted amazon.in for
| you and tried the search query -- looks like it works!
| mrbluecoat wrote:
| +1 for considering how it could have been abused to visit
| blocked sites and circumvent online protections.
|
| The syntax is also intuitive, although site loading seems
| pretty slow but maybe that's just the playground and the paid
| access is much faster.
| marcon680 wrote:
| Yep, we also run all our python code in sandboxes and a few
| other security measures!
|
| Site loading is pretty now slow due to a combination of 1.
| traffic we're currently getting, 2. running a remote
| session, 3. running a few large vision language models, and
| 4. adding waits to allow pages to load/allow you to view
| your search results. We're working on cutting our latency
| significantly since it leads to a better development
| experience.
| desireco42 wrote:
| I have to say, looks both simple and 20 times better then those
| horrible no-code solutions with boxes and arrows.
|
| I think you are onto something here.
| marcon680 wrote:
| Thanks, we hope so! That's an interesting conversation -- I'd
| argue that the graph-based no-code solutions you're referring
| to are for a different set of people than those commonly found
| on HN. As a developer I didn't particularly want to work with
| those tools, so we built this to supercharge our own code-based
| flows instead. I actually don't think the node-based flows are
| horrible at all since they successfully enable non-
| developers/less technical people to use agents/build easy
| automations.
| sprobertson wrote:
| Great demo, straight to the point. It might be nice to have some
| kind of feedback mechanism if it can't find the element on the
| page, or if it's partially cut off. For example, I changed the
| GitHub profile to my own (https://github.com/spro) in the example
| and it doesn't scroll down far enough for the whole image. I
| imagine in general it would be nice to scroll to an element ID
| (or even element description using the vision models) instead of
| a hardcoded value.
|
| Side-note: The comment for the frequency graph is wrong, it
| mentions stars instead.
| marcon680 wrote:
| RE: feedback mechanism -- yep, a feedback mechanism is
| definitely something we're thinking of adding. Since we use
| VLMs that are trained to always output coordinates (i.e., they
| don't have a way to say "not on the page"), we're probably
| going to try fine tuning with some negative examples to see if
| we can build that feedback mechanism in.
|
| One way to hack the scrolling to an element is to first run
| extract_bbox on a natural language description (in your case
| for GitHub it might be "follow button") then take the Y
| coordinate of that element and scroll that number of pixels. I
| just wrote this bit of code that I tested and it brings the
| contribution graph into full view:
| simplex.goto("github.com/spro") coords =
| simplex.extract_bbox("follow button")
| simplex.scroll(coords[1]) simplex.wait(2000)
| image = simplex.extract_image("green tile graph")
|
| But then it incorrectly picks the code review/submissions/etc.
| graph as the green tile graph -- we'll look into it!
|
| re: frequency graph typo -- just pushed a fix, thanks!
| myflash13 wrote:
| Can this use a cloud browser API like browserless?
| marcon680 wrote:
| Yep, any websocket URL works. I see that Browserless offers a
| websocket URL, so you can use them! You just need to pass in a
| Playwright browser object into the Simplex constructor.
| mkagenius wrote:
| VLMs are great - I have been able to use it for a similar project
| too [1]. And it's only going to get better. Congratulations on
| the product launch what's your VLM model for this?
|
| 1. A framework to use/control mobile phones via any LLM -
| https://github.com/BandarLabs/clickclickclick
| marcon680 wrote:
| We finetune our own VLMs for this -- unfortunately prefer not
| to share which ones we use specifically! ClickClickClick looks
| awesome, have you heard of FerretUI
| (https://arxiv.org/pdf/2404.05719)? Pretty similar idea.
| mkagenius wrote:
| Yes, I tried a similar one called "omniparser" - where the
| issue was it was missing annotating some UI elements
| sometimes. Moreover, Gemini and Molmo worked right out of the
| box without needing any fine tune.
| xnx wrote:
| I'm surprised you named your framework clickclickclick instead
| of taptaptap.
| vitalets wrote:
| I've tried the following code:
| simplex.goto("github.com/mitchellh") num_contribs =
| simplex.extract_text("blabla") print(num_contribs)
|
| It outputs all texts from the page. Is it expected? Maybe it
| should fail indicating element could not be found?
| skarpoor wrote:
| extract_text returns all the lines of text it finds within an
| element. Looks like for your case it selected a majority of the
| page from the description, so it's returning most of the text
| on the page.
|
| We currently don't return failure cases (just closest match) --
| but good suggestion! We'll fine tune on some negative cases and
| see if we can catch them
| jackienotchan wrote:
| Would you mind sharing the story behind your pivot from on-demand
| photorealistic vision datasets[0] to browser automation?
|
| [0] https://www.ycombinator.com/launches/Lbx-simplex-on-
| demand-p...
| Mandatum wrote:
| VC funding.
| marcon680 wrote:
| Yeah sure! We mentioned a little bit in the post that we
| stumbled upon this problem when working on a synthetic data
| contract. Internally, we were building software at the time to
| fully automate the process of creating synthetic data, all the
| way from purchasing assets to building scene layouts to
| rendering images. We realized after 2 months of building that
| there wasn't a great need for this problem nor could we build
| something minimal but feature-complete that we could give to
| people and iterate off of. We also saw the writing on the wall
| re: building 3D world models (see WorldLabs and NVIDIA's new 3D
| world models). We're excited to see where this goes!
| TypingOutBugs wrote:
| It's quite a crowded market, browser workflow automation.
| What are you guys trying to do differently? For me, as
| someone who does a bunch of browser automation tasks, the
| real issue is stability. So many tools fall over on non-
| trivial workflows like complex forms with tabs. Fix that and
| you may open up a large testing automation market.
| marcon680 wrote:
| Some of the things we're trying to differently are 1)
| making flows deterministic by using vision models that have
| a consistent output given an image and an input query and
| 2) breaking up flows into these smaller atomic actions in
| order to improve consistency as well.
|
| RE: workflows w/ complex forms and tabs -- do you have some
| sites that are good examples of this? We'd love to see how
| Simplex does.
| globnomulous wrote:
| Apparently a simplex is a concept in geometry. I'm guessing this
| is the intended meaning of the term in the name. When most
| decently well informed, native speakers of English hear
| "simplex," though, they are likely to hear "herpes simplex" and
| so are likely to wonder, as I did, whether, or why, the program
| is named after a sexually transmitted disease.
| internet_points wrote:
| I for one did not make that connection. Only thing I thought of
| was https://simplex.chat/ (which is fortunately not marketed as
| a dating app)
| gashmol wrote:
| I thought it's related to the simplex method in linear
| programming
| TeMPOraL wrote:
| Which IIRC is named for that geometry thingie GP mentioned,
| that this method uses to represent the problem, with the
| solutions being vertices/edges of the thingie.
| globnomulous wrote:
| Interesting, TIL herpes is top of mind for me, but not for
| most of HN.
| IanCal wrote:
| I would _never_ have made that link.
| TypingOutBugs wrote:
| If you gave me the chance to come up with 500 things related to
| simplex, herpes would not be there.
| thasaleni wrote:
| tried: simplex.goto("www.amazon.com")
| simplex.wait(2000) simplex.click('the search bar')
| simplex.type("bicycle") simplex.press_enter()
| simplex.wait(1000) image = simplex.extract_image("first 3
| rows of results")
|
| get error: simplex error processing error
| marcon680 wrote:
| So sorry about this, it looks like our servers went down
| overnight. Just fixed, should be good now!
| replwoacause wrote:
| Nice job! Does this only work with Python? In your docs under API
| reference, I only see 1 endpoint '/find-element'. If I wanted to
| use this via REST API without python, is that possible? Also,
| what kind of pricing can be expected?
| marcon680 wrote:
| Thanks! We're happy to add more REST API functionality -- could
| you email us at founders@simplex.sh if you're up to talk
| further about what you'd like to see? RE: pricing, we're still
| figuring this out!
| is_true wrote:
| Nice.
|
| Does it work with sites that have cloudflare antibot (or similar)
| functions?
| johntash wrote:
| Very cool, any chance of either open-sourcing it or allowing the
| browser part to be self-hosted? i.e. to act on websites hosted in
| a local lan/vpn?
|
| Also, did you evaluate https://github.com/browser-use/browser-use
| by any chance and have any comments about it? I'm assuming it was
| too AI-heavy based on what you said about claude/etc?
| marcon680 wrote:
| Thanks John! We should make that more clear in the docs. You
| can set browser=None on initialization and Simplex will create
| a local browser instance that can run on your local websites.
| We're not planning to be open source right now since a large
| portion of our product is custom vision models + inference
| speedups through hosting.
|
| Browser Use is another YC company. Probably the biggest
| difference is that they're more agent focused while we're more
| lower level -- in the Claude Computer Use camp like you
| mentioned.
___________________________________________________________________
(page generated 2025-01-15 23:02 UTC)