https://github.com/TaxyAI/browser-extension Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + For + Enterprise + Teams + Startups + Education + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} TaxyAI / browser-extension Public * Notifications * Fork 7 * Star 294 Automate your browser with GPT-4 License MIT license 294 stars 7 forks Star Notifications * Code * Issues 4 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights TaxyAI/browser-extension This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 1 tag Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/T] Use Git or checkout with SVN using the web URL. [gh repo clone TaxyAI] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @arcticfly arcticfly Add GitHub flow to README.md ... e098afc Mar 28, 2023 Add GitHub flow to README.md e098afc Git stats * 178 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows tag March 27, 2023 16:29 .vscode remove unused imports on save March 27, 2023 18:17 demos Add shorten_factor parameter to recording-to-gif.sh March 28, 2023 10:03 src more reliably re-enable extensions March 28, 2023 11:50 utils Flatten out browser-extension directory March 25, 2023 21:22 .babelrc Flatten out browser-extension directory March 25, 2023 21:22 .env.example Default debug mode to be off March 27, 2023 13:23 .eslintrc eslint prettier March 27, 2023 18:18 .gitignore Flatten out browser-extension directory March 25, 2023 21:22 .prettierrc Flatten out browser-extension directory March 25, 2023 21:22 .tool-versions Flatten out browser-extension directory March 25, 2023 21:22 LICENSE Update LICENSE March 25, 2023 21:24 README.md Add GitHub flow to README.md March 28, 2023 15:58 jest.config.ts Flatten out browser-extension directory March 25, 2023 21:22 package.json Update version March 28, 2023 10:04 tsconfig.json Flatten out browser-extension directory March 25, 2023 21:22 webpack.config.js Flatten out browser-extension directory March 25, 2023 21:22 yarn.lock eslint prettier March 27, 2023 18:18 View code [ ] Taxy AI: Full Browser Automation Table of Contents Installing and Running Installing the extension Running in your browser How it Works - The Action Cycle Simple Demos Protecting the main branch in GitHub Searching for and playing the movie Oblivion in Netflix Creating a calendar event Writing an essay in the OpenAI playground Add your own! Tech Stack Resources README.md [icon-128] Taxy AI: Full Browser Automation Waitlist | Discord Taxy uses GPT-4 to control your browser and perform repetitive actions on your behalf. Currently it allows you to define ad-hoc instructions. In the future it will also support saved and scheduled workflows. Taxy's current status is research preview. Many workflows fail or confuse the agent. If you'd like to hack on Taxy to make it better or test it on your own workflows, follow the instructions below to run it locally. If you'd like to know once it's available for wider usage, you can sign up for our waitlist. Taxy is fully open-source, and we don't send any page contents or instructions to our servers. Here's Taxy using Google Calendar with the prompt "Schedule standup tomorrow at 10am. Invite david@taxy.ai" calendar Table of Contents * Taxy AI: Full Browser Automation + Table of Contents + Installing and Running o Installing the extension o Running in your browser + How it Works - The Action Cycle + Simple Demos o Protecting the main branch in GitHub o Searching for and playing the movie Oblivion in Netflix o Creating a calendar event o Writing an essay in the OpenAI playground o Add your own! + Tech Stack + Resources Installing and Running Currently this extension is only available through this GitHub repo. We'll release it on the Chrome Web Store after adding features to increase its usability for a non-technical audience. To build and install the extension locally on your machine, follow the instructions below. Installing the extension 1. Ensure you have Node.js >= 16. 2. Clone this repository 3. Run yarn to install the dependencies 4. Run yarn start to build the package 5. Load your extension on Chrome by doing the following: 1. Navigate to chrome://extensions/ 2. Toggle Developer mode 3. Click on Load unpacked extension 4. Select the build folder that yarn start generated Running in your browser 1. Once installed, the browser plugin will be available in two forms: 1. As a Popup. Activate by pressing cmd+shift+y on mac or ctrl+shift+y on windows/linux, or by clicking the extension logo in your browser. 2. As a devtools panel. Activate by first opening the browser's developer tools, then navigating to the Taxy AI panel. 2. The next thing you need to do is create or access an existing OpenAI API Key and paste it in the provided box. This key will be stored securely in your browser, and will not be uploaded to a third party. 3. Finally, navigate to a webpage you want Taxy to act upon (for instance the OpenAI playground) and start experimenting! How it Works - The Action Cycle 1. Taxy runs a content script on the webpage to pull the entire DOM. It simplifies the html it receives to only include interactive or semantically important elements, like buttons or text. It assigns an id to each interactive element. It then "templatizes" the DOM to reduce the token count even further. 2. Taxy sends the simplified DOM, along with the user's instructions, to a selected LLM (currently GPT-3.5 and GPT-4 are supported). Taxy informs the LLM of two methods to interact with the webpage: 1. click(id) - click on the interactive element associated with that id 2. setValue(id, text) - focus on a text input, clear its existing text, and type the specified text into that input 3. When Taxy gets a completion from the LLM, it parses the response for an action. The action cycle will end at this stage if any of the following conditions are met: 1. The LLM believes the task is complete. Instead of an action, the LLM can return an indication that it believes the user's task is complete based on the state of the DOM and the action history up to this point. 2. The user stopped the task's execution. The user can stop the LLM's execution at any time, without waiting for it to complete. 3. There was an error. Taxy's safety-first architecture causes it to automatically halt execution in the event of an unexpected response. 4. Taxy executes the action using the chrome.debugger API. 5. The action is added to the action history and Taxy cycles back to step 1 and parses the updated DOM. All prior actions are sent to the LLM as part of the prompt used to determine the next action. Taxy can currently complete a maximum of 50 actions for a single task, though in practice most tasks require fewer than 10 actions. Simple Demos Protecting the main branch in GitHub main-branch Searching for and playing the movie Oblivion in Netflix Oblivion Creating a calendar event Calendar Writing an essay in the OpenAI playground Playground Add your own! If you have an interesting demo you'd like to share, submit a PR to add your own! Tech Stack Technology currently used by this extension: * Chrome Extension Manifest V3 * React 17 * Webpack 5 * Webpack Dev Server 4 * React Hot Loader * eslint-config-react-app * Prettier * TypeScript Resources * Getting Started with Chrome Extensions About Automate your browser with GPT-4 Resources Readme License MIT license Stars 294 stars Watchers 4 watching Forks 7 forks Releases 1 tags Packages 0 No packages published Contributors 2 * * Languages * TypeScript 83.4% * JavaScript 12.9% * CSS 2.4% * HTML 1.2% * SCSS 0.1% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.