[HN Gopher] I put sheet music into smart glasses [video]
___________________________________________________________________
I put sheet music into smart glasses [video]
https://github.com/kevinlinxc/AugmentedChords
Author : alex1115alex
Score : 180 points
Date : 2025-05-03 01:46 UTC (21 hours ago)
(HTM) web link (www.youtube.com)
(TXT) w3m dump (www.youtube.com)
| alex1115alex wrote:
| One of our users documented projecting sheet music onto his smart
| glasses's display (with a HUD). He did a great job documenting
| the limitations of 2025's tech, but it gives a great look into
| what's going to be feasible next year.
|
| Awesome job Kevin!
| Chipen wrote:
| cool~
| joshuanapoli wrote:
| That's a fabulous project video!
| bix6 wrote:
| Wow can't wait for v2!
| patrickhogan1 wrote:
| That's super cool.
| dang wrote:
| This project is cool so we're hoping to arrange with Kevin to do
| a Show HN about it, so stay, er, tuned!
| swyx wrote:
| wow that's a special honor. is there a way to search up the
| "specially invited" Show HNs?
| dang wrote:
| There's a very incomplete list at
| https://news.ycombinator.com/keyed?k=invited - incomplete
| because it only includes posts that technically got an
| invitation link by email, and there are plenty of others. You
| can also look at https://news.ycombinator.com/pool (explained
| at https://news.ycombinator.com/item?id=26998308) and find
| Show HNs in there. But that list includes many Show HNs that
| we didn't specifically help with beforehand.
|
| (So I guess you can have either a list of too many or a list
| of too few )
|
| p.s. If anyone notices really cool work that would be even
| better if the creator did a Show HN, please let us know at
| hn@ycombinator.com.
| swyx wrote:
| thank you dang!
| swyx wrote:
| fwiw that link is blocked for us: "Please log in as an
| administrator."
| hougaard wrote:
| Fun, did that 8 years ago with the original Hololens
| https://www.youtube.com/watch?v=E6cBX4t2kX0
| vunderba wrote:
| Nice work. From the Github:
|
| _> This allows the pianist to not have to turn pages, and more
| importantly, allows them to see the music and their hands at the
| same time, which is an unavoidable problem with traditional sheet
| music._
|
| I could definitely see this being beneficial for beginners. When
| I lived in a dormitory during uni I often played familiar pieces
| from memory pretty late on a digital piano (with headphones) in
| extremely dim lighting so as not to disturb my roommate.
|
| At some point I just stopped having to look down at the keyboard.
| I play a lot of stride piano as well and that probably
| conditioned me to just have a sort of musical proprioception for
| the instrument. And of course, there's numerous examples of
| unbelievable blind pianists - Stevie Wonder, Ray Charles, Art
| Tatum, etc.
| atoav wrote:
| I wanted to say, after a certain level looking at your fingers
| when playing an instrument becomes the equivalent of looking at
| your legs when riding the bicycle.
|
| When I start to think too much about what my fingers are doing
| I will play worse. For if I want to practise a particular part
| where I get the fingering wrong, sure, but when you play it for
| real, looking is counterproductive.
|
| Something like this could be great for beginners tho. But
| simular to automatic guitar tuners I am not sure if you should
| get into the habit of this technology being around.
| yusina wrote:
| Counter point: people get too hung up on staring at the
| sheet. The sheet is just a tool to help you remember what you
| intend to play. The goal should always be to not need it
| anymore, and while using the sheet, it's like using a crutch.
|
| The "looking at your fingers" challenge then becomes that you
| start to play "by eye" instead of "by ear" (or "by feel")
| which I find is very hard to overcome. Especially when you
| are improvising.
|
| Though in a sense "by sheet" is just as bad.
| schwartzworld wrote:
| > The goal should always be to not need it anymore, and
| while using the sheet, it's like using a crutch.
|
| Uh, why? Lots of pros use sheet music, especially for
| complex pieces. I've never heard of an orchestra conductor
| insisting everyone be off-book.
|
| It's one thing to memorize pop songs or whatever, but
| nobody is out there shaming people for not memorizing
| Rachmaninov
| yusina wrote:
| People who are good at playing a technically hard piece
| essentially know it by heart. They couldn't play it on
| sight, they have invested significant time to be able to
| play it. It's just the few remaining percent of reminders
| and the comfort of having the sheet "just in" case as
| well as the being-used-to-it factor that makes them stare
| at the sheet. But if they can play it really well then
| they don't _actually_ need it. Any pro musician will tell
| you this.
|
| The orchestra setting has the extra requitement that the
| sheet is a tool for communication. "The figure in bar 83"
| is not a term you have gained an intuitive understanding
| for, but is needed to communicate in an orchestra
| setting. The soloist though often times plays by heart,
| at least during performances, so as not to get distracted
| / get tunnel vision.
| necubi wrote:
| The soloist had their concerto memorized (and they've
| probably performed it dozens of times before). But the
| members of the orchestra are responsible for playing
| hundreds of pages of music a week. They've practiced any
| particularly exposed or technical sections, but otherwise
| they're basically sight reading for the performance.
| atoav wrote:
| I am not a big sheet player myself, I love to improvise and
| play by memory, so when I said "don't look at the fingers",
| I didn't mean "instead look at the sheet".
|
| Look at the audience, out of the window, into nothingness
| -- or even close your eyes. As long as you're there.
| bambax wrote:
| > _I just stopped having to look down at the keyboard_
|
| Maybe the next step is an app for people who don't read sheet
| music; it would light up the keys on the keyboard that you need
| to press, when you look at it...?
|
| Same for guitar, highlight where to put your fingers on a fret
| for each chord.
| schwartzworld wrote:
| Various products have done this over the years. Forget app,
| actual keyboards and guitars that light up. It's not a good
| way to learn.
| bambax wrote:
| It's true that it's not a good way to learn, but it's fun
| for a little while.
|
| That's why an app on glasses for this is better than a
| complete alternative instrument; the app is much cheaper
| and should work with any real instrument.
| mock-possum wrote:
| It's not even a good way to play. The delay between seeing
| the cue, and moving to play it, completely ruins the flow
| of playing (and listening to) the music - it introduces too
| many hiccups in timing.
| dylan604 wrote:
| How many people still look at the keyboard when typing? At some
| point, you just don't need to look at whatever it is your
| doing. Also, at some point, you memorize the music if it's
| something you are playing enough. What level of pianist is
| reading sheet music and looking at their hands at the same
| time.
| kevinlinxc wrote:
| For me, it's still easy to mess up for complex sections if I'm
| not peeking at the keyboard every so often. Its true that
| muscle memory takes over after you reach a level of familiarity
| but not quite to the extent of biking
| rtrgrd wrote:
| Cool project! I have a suggestion - since the processing is done
| on a moderately powerful laptop anyway, is it possible to bypass
| the foot pedal and use audio (from laptop or glasses) to predict
| when to switch to the next bar? I assume it will be a complicated
| but would trying to match the FFT series to the sheet music pitch
| data work (or would harmonics cause major headcahes?)
| yusina wrote:
| Music transcription has been around for decades, algorithm-
| wise. In principle this one should actually be easier since it
| doesn't need to transcribe from scratch but "only" find where
| in a given sheet you are most likely to currently be.
|
| (Next step: evaluate afterwords and point out mistakes)
| simonjgreen wrote:
| Super cool PoC. Also an advertisement for the value of local
| processing over cloud.
| OkGoDoIt wrote:
| Now that's an awesome hackathon project! Exciting to see that
| smartglasses are finally getting to an interesting place.
| bambax wrote:
| This is a really cool project.
|
| From the end of the video:
|
| > _I had the bars of music auto-sending at a preset interval. The
| pedals, instead of flipping bars, temporarily pause the flipping
| or speed it up, in case I 'm desynced from the glasses._
|
| That's how teleprompter apps work. Of course the difference is
| that when speaking you can pause a little if you get desynced,
| while with music you're like "on a train" and if you pause, it
| shows. But having an interval is not shocking.
|
| Maybe the problem with this is that typically, sheet music
| resolution is not constant -- if there are many short notes it
| will result in a larger space on the page (a larger bitmap) than
| if there are few long notes.
|
| So maybe an approach is to send a fixed number of bars,
| regardless of their actual size, so that the interval can have a
| constant relation with the tempo of the piece?
|
| > _My dream smart glasses would just listen to the performance
| and automatically flip bars_
|
| Couldn't the phone do that? The phone is already the part doing
| most of the work.
| swyx wrote:
| TIL about their glasses https://mentra.glass/
|
| great video editing, OP. loved the playthru at the end with the
| text. you have real talent here, keep giong
| kevinlinxc wrote:
| Thank you, the typewriter effect is surprisingly hard, requires
| a pretty complicated expression in after effects
| swyx wrote:
| i feel like someone could make "new aftereffects" just making
| the common things that people do easier
|
| got any more of these effects you'd put in that category?
| kevinlinxc wrote:
| Another huge time sink is rotoscoping, e.g. in my video I
| rotoscope the legs of the piano bench and its a cool effect
| but it took almost an hour for those few seconds. AI has
| come far here, but afaik there's no great, free workflow
| yet. Would love to Docker compose something on my computer,
| open a webapp, drag my clip in and export segmented videos.
| Abishek_Muthian wrote:
| Congratulations for the project and for winning the hackathon,
| nicely done!
|
| I am looking for hackable smart glasses with camera which doesn't
| rely upon any proprietary service to work, Mentra seems to have a
| camera version but this video seems to suggest that we need to
| use their service all the time?
| caydenpiercehax wrote:
| Mentra Live is indeed a camera + speakers + microphone pair of
| lightweight smart glasses - that you can build for.
|
| Mentra Live runs AugmentOS, so you can control all the I/O
| (camera, speakers, mic) in your own app with the AugmentOS SDK.
|
| Regarding use the backend service all the time: Most
| apps/developers connect through official AugmentOS.org servers
| and focus on building their apps, but you can self-host your
| own backend if you want.
| ginko wrote:
| Really nice project.
|
| Would be interesting to dive a bit deeper where this 3s latency
| comes from. I assume the bitmaps have been pregenerated so I
| guess it's just the turnaround time when accessing the AugmentOS
| servers?
| kevinlinxc wrote:
| A lot of the time is from the glasses themselves, but
| apparently Mentra talked to the Even CEO and next gen bitmaps
| will be much better
| caydenpiercehax wrote:
| The 3 seconds of latency is writing a bitmap image from the
| phone to the smart glasses over BLE. It's very fast to send
| text, but slow to send bitmaps.
|
| For text, everything is fast - the AugmentOS servers introduce
| <350ms round trip latency in most places/countries.
|
| In next-generation glasses, this will be a lot faster because
| of better bitmaps handling, BMP encoding schemes, and also pre-
| loading of BMPs.
| illwrks wrote:
| This is great!!!
|
| A lot of digital pianos have midi out, (there was a midi
| recording tool posted here months ago by another HN member) I
| wonder if you could use that midi signal to keep what you see in
| sync with your playing to drive page turns? You could even add a
| karaoke like highlight to show the note being played.
| schobi wrote:
| This is really awesome!
|
| I was surprised about using dilation. I would have expected
| music21 to support rendering to a certain resolution/dpi setting
| directly and avoid rescaling the images. But from the music21
| documentation this is not obvious how to do it. Rendering music
| to a low dpi screen nicely (pixel perfect) could circumvent some
| of the hardware limitations in the mid term.
| gavinray wrote:
| I am so unbelievably excited for consumer-grade, useful AR.
|
| There was a lot of hype around VR, but for the last 10 years I've
| been following progress on AR glasses.
|
| The thing about AR is that it has the ability to enhance
| everything in your daily life, versus VR which is meant to be a
| separate experience.
|
| Both Meta and Samsung are due to put out consumer AR glasses
| later this year and I think this might be the first wave of
| useful, daily-wear glasses we'll see.
|
| Is there anyone who works in the AR space that could comment
| more?
| gavinray wrote:
| Just bought a pair of the Mentra Mach1 glasses, let's see how
| they pan out.
| febed wrote:
| Any workaround for those needing prescription glasses?
| alex1115alex wrote:
| You can get prescription inserts from Vuzix, but they're
| pretty bad. If you need a prescription, and want to run
| AugmentOS, your best bet is to buy the Even Realities G1
| instead.
| apples_oranges wrote:
| Unlike you i think the potential is rather in business and
| work. Useful info when needed.
| caydenpiercehax wrote:
| 100% agree.
|
| I've been building smart glasses for over 7 years. First 6
| years were in academia because the tech wasn't ready yet,
| because they were too heavy and battery didn't last long
| enough.
|
| But in the last year, all-day battery smart glasses have become
| lightweight enough to be worn all day (see Even Realities G1,
| Vuzix Z100, etc.).
|
| I believe smart glasses are having their iPhone moment in 2025
| + 2026.
|
| We make the smart glasses OS that Kevin used in the video to
| make this smart glasses app: AugmentOS.org
| _dark_matter_ wrote:
| I want one thing - tell me people's names. I have the unholy
| ability to say your name several times and have an hour long
| conversation, and still not remember it next time I see you.
| People take this so personally that I've started avoiding some
| social gatherings where I only lightly know people! I'd love to
| "know" the name of everyone I've met because I'd be so much
| more comfortable seeing and talking to them again!
| jeffwass wrote:
| Are you me? I have the same issue, when I see someone not in
| my immediate circle and out of context, say on the street, it
| takes me some time to recall their name from my jumbled
| memory.
|
| If there's context (eg I go to another department at work, or
| see my child's friend with their parent) I can get their name
| easier. But that barrier for being out of context can be
| difficult to surmount.
|
| I'm curious if you also find yourself having trouble to
| remembering other names when in conversation (eg what was
| that politician called, what's that city name, it begins with
| an F...)
|
| Usually it's proper nouns that I have trouble recalling. It's
| almost like I need an Anki to refresh my mental DRAM and keep
| things recallable.
| _dark_matter_ wrote:
| Yes I very much have those problems. It's strange because
| I'm objectively a good engineer, but semantic retrieval is
| very difficult for me. It has been my entire life.
|
| When I was a kid I did a round of testing for ADHD, and one
| of the tests is category fluency. Basically they gave me a
| category (e.g. Breakfast foods) and told me to say as many
| words in that category as I could in 60 seconds. I failed
| miserably; the administrator told me it was essentially as
| bad as people with severe cognitive deficiency. I just
| haven't worked about it since then! (Except with names!!)
| mock-possum wrote:
| God yea please - I've wanted this for decades.
|
| Facial recognition, with name, where I know you from, and
| last time I saw you.
|
| Basically I want the same notes my dental hygenist or
| optometrist uses to make light conversation with me during a
| checkup.
| nullhole wrote:
| I'd be neat to have information displayed while driving a car.
| A subset of information currently displayed on a dashboard
| would be an obvious first choice (speed for one).
|
| You could also maybe perhaps tie in a car's knowledge of
| adjacent vehicles, which is something I've wanted for ages.
| Since some newer models have some level of awareness about the
| speed / distance to / relative location of cars around your
| car, you could eg overlay the speed/acceleration info onto
| adjacent cars, so you'd know if you'll need to pass or speed
| up. Seems at least possible since the glasses have awareness of
| your head's orientation, something missing from any existing
| windscreen-style HUD system.
| pazimzadeh wrote:
| Nice, I recently had a very similar idea. I bought the Vusix Z100
| from Amazon japan to do this kind of thing. They also run
| AugmentOS (supposedly? iOS doesn't seem supported yet) so I'll
| try yours out.
| mk_stjames wrote:
| All these glasses have so many layers of abstraction I don't want
| between something I develop and the display.
|
| Let me connect via bluetooth direct to the glasses with anything
| and just tx/rx via a serial port and some low level protocol to
| get pixels/text on the screen.
|
| This is also the only way I'd be able to buy a pair and feel safe
| it won't be able to be bricked in 2 years when some company shuts
| their server down and ends support.
| sintezcs wrote:
| Check out brilliant.xyz
| alex1115alex wrote:
| AugmentOS is open source, so feel free to self-host, or even
| reference our communicator code in our repo to interact with
| the glasses directly via BLE.
|
| This is an inferior means of development, however.
|
| By going through AugmentOS, you get a much easier development
| experience, compatibility with multiple pairs of glasses
| (through both iOS and Android), and the ability to run multiple
| glasses apps simultaneously.
| seabass wrote:
| Really awesome project! I'm reminded of something else in the
| AR/music space from a few years back. Someone made a VR
| passthrough app to project synthesia-style keyboard overlays onto
| your actual piano keys. Always cool to see what new hardware can
| enable. Congrats!
| hhyndman wrote:
| What a great idea. I am a musician and use an iPad for my scores.
| It would be wonderful to replace my glasses with a pair that can
| display the music.
|
| I noticed the iron ring on your pinkie -- Canadian engineer?
| kevinlinxc wrote:
| Correct!
___________________________________________________________________
(page generated 2025-05-03 23:01 UTC)