https://github.com/dcrebbin/meta-vision-api Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} dcrebbin / meta-vision-api Public * Notifications * Fork 3 * Star 76 Hacky Meta Glasses GPT4 Vision Integration 76 stars 3 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights dcrebbin/meta-vision-api This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/d] Use Git or checkout with SVN using the web URL. [gh repo clone dcrebb] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @dcrebbin dcrebbin added the public folder to the repo ... cb2fea7 Nov 28, 2023 added the public folder to the repo cb2fea7 Git stats * 2 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time public src .gitignore README.md bun.lockb env.example package.json tsconfig.json View code Meta Vision Meta Glasses GPT4 Vision API Implementation Setup Get the server up and running: Add the Messenger Chat Observer: Test the integration: Bookmarklet Code Breakdown: README.md Meta Vision Meta Glasses GPT4 Vision API Implementation This is a hacky way to integrate GPT4 Vision into the Meta Rayban Smart Glasses using voice commands. Example Demonstration Requirements: a) Meta Rayban Smart Glasses b) OpenAi Api Key c) Alternative Facebook/Messenger account d) bun Setup Get the server up and running: 1. Add a .env file with your OpenAi API key (example via env.example) 2. Run bun install 3. Run bun run dev 4. Server should be up and running on PORT 3103 Add the Messenger Chat Observer: WARNING: bookmarklets are a slightly obscure and very hacky way to execute arbitrary javascript in your browser, before running MAKE SURE to check the code you're executing. The bookmarklet code is documented below in the section titled: Bookmarklet Code Breakdown 1. Login to messenger.com with an alternative messenger/facebook account (make sure you are friends with your main account that's logged into your meta view app) 2. Copy and paste the code from bookmarklet.js and create a new bookmark in your browser with the URL as the code (alternatively import it as a bookmark) 3. Click the newly created bookmark 4. Upon success a dialog should appear with Added Messenger Chat Observer Test the integration: 1. Make sure within the Meta View app that the messenger connection is connected to the appropriate main account 2. Say You: Hey Meta, send a photo to *name of alternative account* 3. Meta: Send a photo to *name of alternative account* 4. You: Yes 5. Upon receiving the new photo and sending it to GPT4 Vision the server should display the following logs: GPT4 Vision Request Sending request to GPT4 Vision Request Successful Saving data Reading stored data Creating new data file. Writing new data 6. Open up ./public/data.json to check the successful added data ENJOY! Bookmarklet Code Breakdown: javascript: (function (s) { //This a bookmarklet that you can either import as a bookmark //OR you can copy all the code and paste into the URL when making a new bookmark //OR post in dev console // This is designed to observe for any new photo messages that are sent in messenger and then to forward the image url to this projects REST api const messages = document.getElementsByClassName("x78zum5 xdt5ytf x1iyjqo2 xs83m0k x1xzczws x6ikm8r x1rife3k x1n2onr6 xh8yej3")[1].childNodes[2]; // This is to find the messages container within messenger.com for the selected chat // However, these obfuscated classes are subject to change and so this is likely to break in the near future messages.removeEventListener("DOMNodeInserted", null); // The utilization of DOMNodeInserted is very bad practice and will be deprecated in all browsers in the future // Mutation observer should replace DOMNodeInserted messages.addEventListener("DOMNodeInserted", async (event) => { const imgSrc = event?.target?.getElementsByTagName("img")[1]?.src; if (imgSrc) { const res = await fetch("http://localhost:3103/api/gpt-4-vision", { method: "POST", //Facebook's image urls contains lots of properties that need to be perfectly preserved in order to view the image body: JSON.stringify({ imageUrl: imgSrc }), mode: "no-cors", headers: { "Content-Type": "application/json", }, }); const data = res.json(); console.log(data); } }); alert("Added Messenger Chat Observer"); })(); by Devon Crebbin Please reach out if there are any issues or feature requests :) Hopefully the Meta Reality Labs team will provide an SDK in the future so these types of integrations can be productionised About Hacky Meta Glasses GPT4 Vision Integration Topics meta gpt4 Resources Readme Activity Stars 76 stars Watchers 2 watching Forks 3 forks Report repository Releases No releases published Packages 0 No packages published Languages * TypeScript 83.6% * JavaScript 16.4% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact You can't perform that action at this time.