[HN Gopher] Show HN: Hacky Meta Glasses GPT4 Vision Integration
___________________________________________________________________
Show HN: Hacky Meta Glasses GPT4 Vision Integration
Super hacky implementation due to the lack of an SDK. Fun project
though. In the foodlog demonstration I just made a fake fb account
(sorry zucc) called "Mye Food-Log".
Author : devon_c
Score : 188 points
Date : 2023-11-29 10:43 UTC (12 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| amne wrote:
| somewhat off-topic but this is the first time I see bun actually
| used for something other than "let's compare it to node"
| ge96 wrote:
| Someday, witty remark about anaconda and bun
| devon_c wrote:
| I generally just follow the "It's a drop in replacement for
| node" and hope the APIs I need are there. Also makes me feel
| like a 2023 bleeding edge blazingly fast developer which is 90%
| of the use case right?
| pzmarzly wrote:
| The project seems to have 0 external runtime dependencies, it
| should work without "bun install". That is really cool. In the
| code, it is using typescript, fetch API (so no need for node-
| fetch) and Bun.serve (so no need for express).
|
| From my experience, a lot of users fail to get Node.js projects
| running - the distributions (Debian, Ubuntu) still ship ancient
| 10.x or 12.x versions, npm may or may not be installed, "npm
| install -g" will try writing to /usr by default and fail with
| EACCESS, etc. Bun gets the UX right - I think that author could
| even try "bun build --compile" to get ELF binary with bun
| interpreter and JS payload.
| soderfoo wrote:
| Awesome demo, short simple and to the point.
|
| I did not know Meta released smart glasses. Has anybody found
| them useful or is it more of a novelty?
| devon_c wrote:
| They're actually good and have nearly replaced my airpods. I
| don't need glasses and don't live in SF so I do feel a bit
| pretentious though.
|
| The potential is definitely there and I'd be very surprised if
| the Meta Reality Labs team haven't already added heaps of
| features internally and are just waiting or staggering
| releases.
| TheHumanist wrote:
| Ok, so how long have you had these? I also was unaware they
| had released these but they look pretty fun AND the price
| point is surprisingly approachable.
|
| Are you wearing these out in public pretty regularly? Are
| people noticing the lenses? My main concern with buying these
| is that I will end up not wearing them in public eventually,
| which kind of makes them not super useful. I worry about that
| because I can see some people seeing those lenses and going
| off on the whole bit about am I filming them? am I filming
| their kids? And we all now the rabbit hole of uncomfortable
| public social iteractions this can take one down. So, I worry
| I would run into too many of these people downtown or
| something and end up being anxious about even wearing them
| out anywhere.
|
| What is your experience and how long have you been wearing
| them around in public (if you are)?
| devon_c wrote:
| Wore them everyday for 2 weeks and I actually only bought
| them as I thought I could make something like this (through
| any hacky means).
|
| No one has noticed they're any different, especially
| outside as I got transition lenses so they just go into
| normal rayban sunglasses mode. You could have a similar
| effect if you wore those old "new" 3D glasses out and
| about.
|
| Even colleagues had to squint.
|
| If I do ever have a bad interaction with strangers I'll
| just tell them about the lengths meta have gone through to
| prevent bad actors. (i.e: knowing you're trying to cover up
| the flash).
| TheHumanist wrote:
| What has Meta done to prevent bad actors?
| devon_c wrote:
| No SDK to hack (yet?), bright flash, loud "shutter" sound
| and anti flash coverup.
| swyx wrote:
| the sound quality isnt as good as airpods though!
| KaiserPro wrote:
| I have the previous version.
|
| They make brilliant headphones. They have good microphones so
| they also work a great hands free, perfect for driving.
|
| The cameras are subtle, and only get noticed when you point
| them out to people. In the street you're not going to notice
| they are different from any other wayfarer.
|
| I don't use the assistant, because I'm never going to say "hey
| facebook" in public. Nope, fuck that.
|
| the video is smooth, even if you are running. the stills are
| kinda useful for "in the moment" pictures.
|
| For the newer version, I think its "hey meta" and they appear
| to do a lot more than just allow you to read facebook messenger
| messages.
| digitcatphd wrote:
| Really cool and actually a real use case.
| laputan_machine wrote:
| Very cool! Hacker ethos (make a fake account to leverage sending
| photos), short and simple demo. Nice job
| devon_c wrote:
| Thanks!
|
| I first attempted a more standard approach via a webhook hosted
| on Heroku linked to an fb page, which theoretically would of
| worked.
|
| But, Meta has limited you to only sending messages/pictures to
| "real people".
|
| I believe they have to be on your friends list for a certain
| amount of time too but haven't properly tested all the edge
| cases.
| TheHumanist wrote:
| Oh is this your project? It's really cool!
|
| Ok, so how long have you had these? I was unaware they had
| released these but they look pretty fun AND the price point
| is surprisingly approachable. Are you wearing these out in
| public pretty regularly? Are people noticing the lenses? My
| main concern with buying these is that I will end up not
| wearing them in public eventually, which kind of makes them
| not super useful. I worry about that because I can see some
| people seeing those lenses and going off on the whole bit
| about am I filming them? am I filming their kids? And we all
| now the rabbit hole of uncomfortable public social
| iteractions this can take one down. So, I worry I would run
| into too many of these people downtown or something and end
| up being anxious about even wearing them out anywhere.
|
| What is your experience and how long have you been wearing
| them around in public (if you are)?
|
| Also, legit question... how did you succesfully make an alt
| facebook account? lol Anytime I try with a different name so
| I can have an alt for buying stuff on marketplace it blocks
| that new account out because I can't verify it is me (because
| it's a fake name).
| anonzzzies wrote:
| How about for the quest 2/3? Anyone did that?
|
| What similar quality vision models are there outside gpt4 ? It's
| annoying it has so maybe (albeit mostly hack-prompt-able)
| restrictions.
| Philpax wrote:
| No camera access on those, unfortunately.
| anonzzzies wrote:
| Ah didn't know that, thanks.
| DevX101 wrote:
| I love this demo! 35 seconds, terrible video quality, but it
| shows you can unlock so much more. Great job.
| devon_c wrote:
| Thanks!
|
| Recorded on OBS, uploaded to linkedin, screen recorded on iOS
| then uploaded to Youtube.
|
| Is that not a standard workflow haha (initially was just to
| send to 1 mate who didn't have linkedin lol)
| webappguy wrote:
| Sorry did I miss something in the demo, I must have as it
| appeared you just took a photo and saved it, a function
| native to the glasses. Where did the vision come in? I'll
| watch again. Or was it that you saved it to a feed and not
| the native app
| meesles wrote:
| Where did the details about his food item come from in the
| list? That was GPT4 turning his vision into data.
| _1 wrote:
| He's created another Facebook account. When he takes a
| photo, he tells the glasses to share it with his other
| account. He has a service that checking for messages sent
| to that account. When a new message arrives, theres GPT4-v
| script that analyses the image and logs the results in his
| food tracker.
| jcims wrote:
| This is awesome! I got my glasses a couple weeks ago. I'm in
| Vegas and actually wore them for the first time yesterday.
| They're actually really nice and i was thinking how cool it world
| be to hook it into gpt4v.
| _giorgio_ wrote:
| What "feedback" do you get from the glasses. Imagine taking a
| picture or sending a message...
| _1 wrote:
| Audio only.
| TheHumanist wrote:
| Ok so do you just wear these around in public? Do you think
| people will see the pretty obvious lenses and be
| confrontational? That's the only thing stopping me... I love
| tech like this but I'm not trying to get bitched at in public
| when people think I'm filming them or their kids or something
| like that, you know?
| KaiserPro wrote:
| Its really hard to notice that they are different from normal
| raybans, unless you take a picture.
|
| The lens are pretty subtle, and they look like normal
| wayfarers. When the light comes on, as you make a recording
| or take a picture, then they notice, but not all of the time.
| TheHumanist wrote:
| Ok, so if I wanted to walk through downtown on a pretty day
| and livestream that, it will have a bright light to let
| people know I am recording? I can't just do that subtly?
| KaiserPro wrote:
| Yeah if you are recording, the new ones have a bright
| light (possibly flashing I'm not sure) on. Which makes
| sense really.
|
| They offer better protection against "unsuspecting
| picture taking" than normal phones, as the light is
| vaguely tamper proof. I'm not sure they record if you
| take them off either.
| mistermann wrote:
| I would think some paint could solve that problem or
| would that be detected?
| vlovich123 wrote:
| Or a needle to pop the light.
| KaiserPro wrote:
| The detection is pretty good.
|
| but, my friend, why do you want to cover up the light?
| post_break wrote:
| I've worn GoPros around live streaming and people don't
| notice. Wearing these glasses with a little led on the
| frame 99.9% of people won't notice.
| Manouchehri wrote:
| I have the first gen Ray-Ban Stories (they look very
| similar), and almost nobody noticed (~3 people in 2 years). I
| have clear lenses and wear them like normal glasses every
| day. I mainly use them for the speakers for notifications.
|
| When the new Meta smart glasses were announced, I suddenly
| had >6 people comment on my old glasses. All the comments
| were positive, but it did make me more aware that I might get
| a negative reaction in the future.
| andyjohnson0 wrote:
| Have you tried wearing them in a casino?
| Manouchehri wrote:
| Shouldn't be any different than holding your phone out in a
| casino.
| michaelbuckbee wrote:
| Do casinos ban smartglasses?
| AlecSchueler wrote:
| When Google Glass was around there was a big social pushback.
| Have we accepted the use of these devices around ourselves since
| then?
| ConfusedDog wrote:
| I was not gonna spend $1500 for Google glasses, but $300 for a
| Rayban isn't bad.
| AlecSchueler wrote:
| My understanding was that the pushback was based on people's
| discomfort around other people wearing these things, hence
| the "glasshole" name being used.
| Janicc wrote:
| I think the pushback will be pretty insignificant simply based
| on the fact that compared to Google Glass, they really don't
| stand out as much.
| AlecSchueler wrote:
| Doesn't that make the privacy concerns that people previously
| had even more prominent?
| crims0n wrote:
| Do you have an expectation of privacy in public?
| AlecSchueler wrote:
| I have an expectation that people I'm speaking with
| aren't filming me without making it clear that they're
| doing so. Anyone doing otherwise I consider a creep and
| would rather not interact with.
|
| Do also note that people wear glasses in private settings
| as well as public.
| nickthegreek wrote:
| The glasses have a light on when recording. The light
| cannot be disabled.
| AlecSchueler wrote:
| Your can't draw over it with a marker? What about the
| microphones? Will this still be the case in the next
| generation or in glasses produced by other manufacturers?
| nickthegreek wrote:
| No, if you cover the light, it will refuse to take photos
| or videos. Who cares about the next glasses from another
| manufacture? We are talking about the current Meta
| glasses that just came out.
| AlecSchueler wrote:
| I care and presumably other people with the same outlook
| care, that's why I was asking.
|
| You're trying to assuage my concerns with the information
| about the light. That's Meta responding to these concerns
| and taking efforts to avoid upsetting anyone.
|
| But once the market is proven then another set of glasses
| will be marketed as having the feature of silent
| recording. By that time the force of the market will be
| too great and my concerns will be laughed at and I'll be
| called a luddite and told I never should have had an
| expectation of privacy to begin with.
|
| This is the embrace before the inevitable extinguish.
| nickthegreek wrote:
| The only solution to your concerns are legislation, and
| that is not going to happen in America as everyone has a
| video camera in their pocket and the 1st amendment
| exists. Private establishments are free to make rules
| regarding the use of these on their grounds assuming that
| the user is not using them for a disability related
| service.
| AlecSchueler wrote:
| I'm aware of that of course. No one can turn the tide
| against the market. But why do they bother, then, with
| the whole show of making it only record with a light? You
| agree that that's just theatre to seem less invasive than
| they inevitably will be?
|
| I'm not based in the US so hopefully some local laws will
| help me out a bit, but at the end of the day, the law is
| not my moral barometer. I will respect the rights of the
| users of these devices while also exercising my own
| rights, as much as I am able to, to treat them as social
| pariahs.
| cal85 wrote:
| I think you are very confused about what a reductio ad
| absurdum is. It is not a fallacy, it's a perfectly valid
| form of argument (similar to a 'proof by contradiction'
| in maths), and it doesn't seem to be what you mean here.
| AlecSchueler wrote:
| Thanks, I'll look into it and update my comment.
| TheHumanist wrote:
| But aren't the lenses more obvious on these? Just looked at
| the Glass and I guess they are about as obvious as the lense
| on that but the glass stands out more for sure because of its
| design (which I loved).
| itsyaboi wrote:
| These look like ordinary sunglasses while google glasses
| were akin to a dragonball z accessory.
| etrautmann wrote:
| IMO the pushback was against the perception that people wore
| them to signal that they had $1500 since they didn't do that
| much and were fairly obviously futuristic on your face. I saw
| someone climbing in them and my impression was not that they
| were somehow relevant for the climbing..
| Angostura wrote:
| The push back was largely about the creepiness of being
| potentially surveilled all the time. This is just the same
| floren wrote:
| yeah it would be pretty fucked up if I was recorded all the
| time, like in every store. Or like walking through a
| neighborhood and half the doorbells call out as you pass to
| inform your that they're recording. Or if you do anything
| slightly embarassing in public and there is a teenager
| nearby.
| AlecSchueler wrote:
| We can accept the trade offs for some surveillance while
| still being critical of new trends. Or maybe you're ready
| to have a 24/7 livestream from your toilet?
| floren wrote:
| The toilet came up constantly in discussions of Google
| Glass, as though the presence of a camera on your head
| would suddenly turn you into a raving lunatic who pops
| his head over bathroom stalls and peeks over the urinal
| dividers.
| AlecSchueler wrote:
| I'm not sure I saw that argument at the time and I'm not
| making it now either. I don't know what to do with this
| response.
| TheHumanist wrote:
| This is my main concern. These lenses are much more obvious
| than the Google Lense cameras as well. I'd love to use some
| cool tech like this but I can absolutely see some people seeing
| the lenses and being confrontational and trying to act like you
| are purposefully filming them, or their kids... or something
| along those lines. Yes, it is legal to film in public but I'm
| not trying to have those sorts of confrontational interactions.
| 6gvONxR4sf7o wrote:
| Unfortunately the lesson some people learned from google glass
| was 'be sneakier.' Hence the design of these.
| m00dy wrote:
| Obviously this is the future just need a bit more powerful tech.
| criddell wrote:
| When I see demos like this I think of William Gibson who said
| "The future is already here - it's just not evenly
| distributed."
| alonsonic wrote:
| This is a fantastic demo. I've tried to build these type of
| prototypes for all Meta Headsets in the past but their very
| limited API/SDKs block you from doing anything meaningful with
| computer vision. They are too scared of devs getting access to
| the camera.
|
| Hope that Apple Vision Pro gives a more robust api to developers
| and that forces Meta to open up.
|
| I will use this approach for a poc I have in mind. Great job and
| thank you for open sourcing!
| _giorgio_ wrote:
| No jailbreak yet.
| devon_c wrote:
| I did recently try to reverse engineer the connection Instagram
| are doing with the glasses in order to livestream through the
| glasses.
|
| However, not too familiar with trawling through decompiled APKs
| (also presume there's some sort of internal secret they're
| using too)
|
| No worries!
| fullspectrumdev wrote:
| That's been on my endless todo list since I got the first
| version of the View glasses from FB/Meta ages back.
| yunohn wrote:
| > Hope that Apple Vision Pro gives a more robust api to
| developers
|
| I was under the impression that Apple is keeping camera access
| completely locked down?
| procgen wrote:
| Third-party apps can access a single composite "front
| camera", but only if a "spatial persona" is found on the
| device.
|
| https://developer.apple.com/videos/play/wwdc2023/10094/?time.
| ..
| codeulike wrote:
| Hey Meta, send a photo to Eez Discheaperon Amazon
| WhitneyLand wrote:
| Nice work Devon!
|
| It may be "super hacky" as you say, but these projects are
| important.
|
| We may have had a theoretical understanding that this was
| possible, but seeing your demo opens up our experiential
| understanding.
|
| Lights up totally different parts of the brain.
| johnwheeler wrote:
| I was thinking the same thing. Obviously this is not meant to
| be a polished product, but it lights a spark that will
| eventually grow into a fire. We've always wanted this
| capability, and this guy has shown us it practically possible
| now. Doubtless there will be others, but that's the point.
| fgblanch wrote:
| Very nice hack! I did a very similar project integrating ChatGPT
| bot but using WhatsApp business account instead of fake facebook
| contact. I got my account blocked when Meta discovered I'm not a
| business unfortunately. I'll retake the project with the FB
| account, it seems much easier.
|
| Great job!
| stuckkeys wrote:
| I will wait for some kind of version that does not require a
| facebook account. Cool demo tho.
| charcircuit wrote:
| It does not require a facebook account. It uses a separate meta
| account.
| b8 wrote:
| Matter of time until someone hacks Meta's glasses to cheat on
| tests.
| padjo wrote:
| Bookmarklets are still a thing? I just sorta assumed browsers
| shut them down years ago
| gourneau wrote:
| Meta folks if you are reading this , please give us a real SDK.
| Great hack though until we get that!
| paul7986 wrote:
| What have others experience been using Hey Meta with their Meta
| glasses? Myself not great as it feels like a beta feature..
| doesn't run reliably. Tho good effort by them using current
| technology.
|
| What the glasses do now reliably are taking photos, short videos
| and transferring them into the Meta View app. Bluetooth audio
| transferring from my car stereo into the glasses is annoying and
| I couldn't find a setting to change / stop that.
|
| As for Meta Vision there's so much innovation to happen here,
| nice!! gives me tons of ideas
___________________________________________________________________
(page generated 2023-11-29 23:01 UTC)