[HN Gopher] Synchronizing Pong to music with constrained optimiz...
___________________________________________________________________
Synchronizing Pong to music with constrained optimization
Author : platers
Score : 191 points
Date : 2024-09-03 13:19 UTC (9 hours ago)
(HTM) web link (victortao.substack.com)
(TXT) w3m dump (victortao.substack.com)
| randall wrote:
| pretty neat! it feels like if you spaced out "important" beats
| instead of most of them and shrunk the play area so the paddles
| are larger, it would have an even more interesting effect.
| SamineDylah wrote:
| Absolutely wonderful!
|
| > "We obtain these times from MIDI files, though in the future
| I'd like to explore more automated ways of extracting them from
| audio."
|
| Same here. In case it helps: I suspect a suitable option is
| (python libs) Spleeter (https://github.com/deezer/spleeter) to
| split stems and Librosa (https://github.com/librosa/librosa) for
| beat times. I haven't ventured into this yet though so I may be
| off. My ultimate goal is to be able to do it 'on the fly', i.e.
| in a live music setting being able to generate visualisations a
| couple of seconds ahead being played along with the track.
|
| Not sure if this is unsavory self promotion (it's not for
| commercial purposes, just experimenting), but I am in the middle
| of documenting something similar at the moment.
|
| Experiments #1 - A Mutating Maurer Rose | Syncing Scripted
| Geometric Patterns to Music:
| https://www.youtube.com/watch?v=bfU58rBInpw
|
| It generates a mutating Maurer Rose using react-native-svg on my
| RN stack, synced to a music track I created in Suno AI *.
| Manually scripted to sync up at the moment (not automatic until I
| investigate the above python libs).
|
| Not yet optimised, proof of concept. The Geometric pattern (left)
| is the only component intended to be 'user facing' in the live
| version - But the manual controls (middle) and the svg+path html
| tags (right) are included in this demo in order to show some of
| the 'behind the scenes'.
|
| Code not yet available, app not yet available to play with. Other
| geometric patterns in the app that I have implemented:
|
| - Modified Maurer
|
| - Cosine Rose Curve
|
| - Modified Rose Curve
|
| - Cochleoid Spiral
|
| - Lissajous Curve
|
| - Hypotrochoid Spirograph
|
| - Epitrochoid Spirograph
|
| - Lorenz Attractor
|
| - Dragon Curve
|
| - Two Pendulum Harmonograph
|
| - Three Pendulum Harmonograph
|
| - Four Pendulum Harmonograph
|
| This is the Typescript Maurer Rose function (that is used with
| setInterval + an object array of beat times which determine when
| to advance the 'n' variable): export const
| generateGeometricsSimplemaurer = (n: number, d: number, scale:
| number = 1) => { const pathArray: TypeSvgPathArray =
| []; for (let i = 0; i <= 360; i += 1) {
| const k = i \* d; const r = Math.sin(n \* k \*
| (Math.PI / 180)); const x = r \*
| Math.cos(k \* (Math.PI / 180)) \* 40 \* // base
| scale scale + 50; // to
| center the image const y = r \*
| Math.sin(k \* (Math.PI / 180)) \* 40 \* // base
| scale scale + 50; // to
| center the image pathArray.push(\${i === 0 ? "M" :
| "L"} ${x} ${y}`);` } const pathString: string
| = pathArray.join(" "); return pathString; };
|
| setInterval isn't an appropriate solution for the long term.
|
| The geometric patterns (with their controls) will have a
| playground app that you can use to adjust variables... As for the
| music sync side, it will probably take me a long time.
|
| *Edit: I just noticed that the author (Victor Tao) actually works
| at Suno
| entropie wrote:
| Really nice stuff. I cannot send a heart without subscribing
| which doesnt feel right for me.
| tongbaojia wrote:
| Awesome work bro! Your company is hiring? I'd be super thrilled
| to work with you.
| adroitboss wrote:
| This is so freaking cool! I was mesmerized watching the paddles
| move as the beat progressed. There are certain things that just
| look right which makes it beautiful.This project is one of them!
| jfmc wrote:
| Prior art: Eisenfunk - Pong
| (https://www.youtube.com/watch?v=cNAdtkSjSps)
| diggan wrote:
| Bit different though! In your example, the video is made from
| manually syncing with the song bpm, as the beep is at a
| constant rate. It's basically just a hand-made visualization of
| (every other) kick drum.
|
| While the submission has the notes not at a basic 1/4 tempo,
| and is automatically "animated" based on the constrained
| optimization. Also leads to a much more interesting
| visualization :)
| jfmc wrote:
| No constraint optimization can replace Pentafunk Jenny ;)
| ChrisMarshallNY wrote:
| I love that video. Weird, but catchy.
| KolmogorovComp wrote:
| Awesome work!
|
| How is the beat used to sync the pong chosen? Like for Bad
| Apple!, especially around 1m55
| https://www.youtube.com/watch?v=bvxc6m-Yr0E it seems off
|
| Good suggestion from a YouTube commenter, pasting it here
|
| > This is pretty cool.. it would be cooler if there were multiple
| pongs and paddles for each type of beat (like high beats and low
| beats)
| montebicyclelo wrote:
| Very cool! As a further variation on this idea, I'm imagining
| training a reinforcement learning agent on atari games / super
| mario, but with an additional music-based reward/input, to try to
| get a "musical" looking playthrough... (Not sure how good it
| would look / whether it would be worth it though...)
| vunderba wrote:
| Crypt of the NecroDancer explores this idea of rhythmically
| timing your character's movement to get bonuses in game.
| Sirizarry wrote:
| Don't forget the legend of Zelda spinoff "cadence of hyrule"
| which I'm pretty sure was made by the same guys as crypt of
| the necrodancer.
| jerf wrote:
| Bit.Trip Beat is a game based on almost exactly the concept
| in the linked page: https://youtu.be/LHbg-sNqe4w?t=47 , as
| well as its later sequel Flux.
| dberst wrote:
| I'm a novice at machine learning, but Open AI made a python
| library for reinforcement learning in video games, and a fork
| of it is still actively maintained [1]. It's been a few years,
| but I remember being able to get it up and running in a day or
| two (maybe a weekend). It used the Retroarch emulator, which is
| compatible with a huge number of emulators and consoles.
|
| https://github.com/Farama-Foundation/Gymnasium
|
| There also SethBling's excellent video on YouTube about machine
| learning specifically with Super Mario World:
|
| https://www.youtube.com/watch?v=qv6UVOQ0F44
|
| I encourage you to give it a try! I feel that video games are a
| bit underrated by current AI buzz, I think there's lots of
| potential for a machine to learn a skill through playing a
| game. And lots of potential for a game being selected or even
| created with the goal of teaching a specific skill. However at
| that point maybe it's better to forego the audio and visuals
| and speak to the machine in text or with pure data.
|
| On the other hand, I have seen a video about convolutional
| neural networks that feed on each pixel of an image. So perhaps
| training with sound data, or with the pixels of a spectrogram,
| could have some positive results. And certainly I would be
| amused to see a game played in time with music, or possibly
| even dancing with a songs melody, harmony, and story as well.
|
| Anything that's ever been created by humans, existed first in
| the imagination of a human brain. And you've got one of those.
| A mental vision pursued and brought from the mind into physical
| reality is a beautiful thing, a gift for all of humanity in my
| eyes. I think it's quite worthwhile. But that's just my
| perspective. Thank you for sharing your imagination. Have a
| nice day
| grimgrin wrote:
| Imagining an `installation` in my space, using both my MT-80S and
| a display. Can I even reason about this, the timing? I'm not
| smart here, just interested
|
| https://www.matrixsynth.com/2014/07/roland-mt-80s-midi-playe...
| Angostura wrote:
| Really interesting. For some reason my brain really really hates
| this. I think it screws with my internal model of causality or
| something and I find it difficult to watch. Odd
| johntb86 wrote:
| It seems like the ball bounces off the center of the paddle,
| not the edge, which always makes it look wrong. Maybe you're
| seeing the same problem?
| dwringer wrote:
| For me the most conspicuous thing missing is dynamics...
| particularly when there are "ghost notes" in between much
| louder ones during a fast passage, it seems like something is
| missing.
|
| That said, however, I find it to be oddly satisfying to watch.
| Curious if experience playing different instruments has
| anything to do with it. To me, something like a xylophone or a
| steelpan feels pretty analogous to this.
| fisherjeff wrote:
| I think it may be synthesized - I thought the reverb was a
| little off, and noticed there was little to no change in
| timbre when the dynamics changed.
| MeteorMarc wrote:
| Delightful. Part of the fun is that the game is a background to
| the music rather than the other way around that we are used to.
| tmnvix wrote:
| > Synchronizing pong to music with constrained optimization
|
| Nothing new. Apparently there are references to people doing this
| in ancient and medieval times.
|
| https://en.wikipedia.org/wiki/Flatulist
| bgoated01 wrote:
| This is really cool. I took an optimization class a few years
| back, but haven't made the time to do anything fun with it since.
| This inspires me to do it.
|
| I do kind of wish that the last note corresponded to a game over,
| though, and I wonder if a smaller screen or faster ball would
| widen the playing field a little. Maybe I'll fork the code and
| try some of those out myself.
| jcims wrote:
| This reminds me of those polyrhythm visualizations on YouTube
| (check out LucidRhythms for some great examples).
|
| https://www.youtube.com/@LucidRhythms
|
| Probably almost impossible to adapt written works 'backwards'
| into a visualization but it might be fun to have different bars
| represent different notes and have the balls split for chords.
| cesaref wrote:
| This reminds a bit of a project I coded the audio for an art
| exhibition around 20 years ago, which was for multiple
| simultaneous players. As the game got more intense it became
| apparent that the ball/wall sounds were playing music, and that
| the three players were all actually playing one musical piece.
|
| It was based around 3 arcade cabinets pointing together, so the
| players couldn't see what was on each others screens.
|
| This was achieved by modifying the ball speed/direction slightly
| so that it arrived at the bat/wall at a musically relevant point
| and triggered the correct sound.
|
| Ah, here you go, Josh as a reference to it on his site:
| https://www.autogena.org/work/ping
| rmnclmnt wrote:
| If only operational research courses could have been so fun many
| years ago... excellent write-up!
___________________________________________________________________
(page generated 2024-09-03 23:00 UTC)