[HN Gopher] Bark - Text-prompted generative audio model
___________________________________________________________________
Bark - Text-prompted generative audio model
Author : obi1kenobi
Score : 284 points
Date : 2023-04-20 17:23 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| awaller1 wrote:
| this is incredible
| mysterybox wrote:
| WTH1!!!
| newswasboring wrote:
| Man I know this is HN, and I know we have a certain decorum we
| should be maintaining, but with the recent activity in this field
| the most appropriate response to these posts is "4bit when?" or
| "f16 when?". Not sure which one is applicable. I am having no
| luck running it on a 6GB vram gpu, so I guess its the 16 bit
| floating point one.
| montebicyclelo wrote:
| related to this - to those releasing models, it would be great
| if you could share how much VRAM is required (seems very common
| for this key piece of info to be missing).
| joseph_grobbles wrote:
| I'm successfully running it on a 12GB GPU (while it downloads
| some 12.1GB of model data on first run, the highest GPU memory
| usage was ~6.5GB, settling back down to around 5GB), however
| the results are nothing like the samples given on the github
| page. Using the exact code given and in the runs I've tried the
| results are rather terrible.
|
| I'm not being negative -- some of the samples are really neat
| on their page -- and I know there is some idiosyncrasy of my
| setup that is causing issues, though it is a pretty typical
| conda + pytorch with CUDA 11.8.
|
| Playing with the text and waveform temp from their defaults 0.7
| is yielding some semi-decent results, but it feels essentially
| random.
| sschueller wrote:
| "However, to mitigate misuse of this technology, we limit the
| audio history prompts to a limited set of Suno-provided, fully
| synthetic options to choose from for each language."
|
| Isn't this open source and can be easily removed or am I missing
| something?
| gs17 wrote:
| Yes, it seems to be enforced by a few assert statements in the
| code.
| txtai wrote:
| Excellent work! Hope to see a version that has a friendlier
| commercial license in the future (current version is CC-BY 4.0
| NC).
| jdprgm wrote:
| It seems like a lot of the entries in TTS are either close
| sourced saas apps or something like this with limitations on
| customizing it. It seems clearly inevitable and likely only
| months away that a high quality unrestricted open source option
| for things like voice cloning will emerge so i'm not sure why
| these projects are even really bothering trying to stop it. I
| think in order for TTS to have its StableDiffusion moment it will
| just be a matter of an unrestricted easily trainable open source
| model.
| kleer001 wrote:
| >> i'm not sure why these projects are even really bothering
| trying to stop it.
|
| CYA aka https://en.wikipedia.org/wiki/Cover_your_ass
|
| also it still requires tons of money to run, so it's likely
| only businesses will do it
| rck wrote:
| Any idea what the training data for this is? Looking at the
| model, it looks like it is literally just copy-paste from
| Karpathy's nanoGPT, so the training data is what's most
| interesting. Pretty amazing anyway.
| unraveller wrote:
| I found a secret demo page that shows in real time how they
| assess any sound file's mood swings along with number of
| detected laughs, coughs, etc. Guessing that ability is involved
| somehow.
| gkucsko wrote:
| haha https://demo.suno.ai
| pfa344 wrote:
| Hello how are you doing
| 101008 wrote:
| The Spanish example (Miguel) is really bad.
| gkucsko wrote:
| it's more meant to show code switching. more examples here:
| https://suno-ai.notion.site/Bark-Examples-5edae8b02a604b54a4...
| wtk wrote:
| Polish one very realistic on the other hand
| Tade0 wrote:
| Just about the only unrealistic thing is it recommending
| Szczecin's old town.
| calny wrote:
| Very cool. Side note: bark-gpt.com is already taken for a dog
| translator: "The world's first AI powered, real-time
| communications tool between humans and their furry best
| friends."[0] I only know this because my law firm partner's name
| is Bark, and I wanted to automate some legal work and name the
| software "Bark GPT" after him.
|
| [0] https://www.bark-gpt.com/
| cm2187 wrote:
| It's genius idea. As long as you tell owners what they want to
| believe their pet says, those guys will make a fortune.
| tough wrote:
| I already know when my dogs needs to eat, drink, shit or pee
| or go for a walk or play because he usually will tell me.
|
| Choosing to ignore your dog won't change because some magical
| AI can now translate it to -Im fine, Im only barking because
| you're an awesome being, keep your subscription humaaan-
| sangnoir wrote:
| But can your dog (translator) say "I love you?" In a doggy-
| voice? Replika proves people will pay for this and convince
| themselves it's real, because they want it to be.
|
| Bark-GPT's VC pitch: "Replika for real dogs"
| dasickis wrote:
| This is an April Fool's joke[1] and really good! We're doing
| this for real: https://sarama.app.
|
| Reference:
|
| 1. https://www.laika.berlin/en/blog/new-client-barkgpt-ai-
| dog-b...
| lIl-IIIl wrote:
| Usually I can tell whether something is a parody/joke website,
| but here I am struggling.
| seydor wrote:
| I want to see the training set for this
| coolspot wrote:
| Woof!
| miki123211 wrote:
| The fact that this is open source and can generate more thann
| just speech is really nice, but for speech itself, it's much
| lower quality than what Eleven Labs provides.
|
| All the open source models I've seen so far have this weird kind
| of neural fuzziness to them. I don't know what Eleven does
| better, but there's definitely a big difference.
| drowsspa wrote:
| Seems like it's doable to fix it in post, but I guess nowadays
| we're all about just shoving everything into the model
| ignoramous wrote:
| Bark's readme points out that to access the "larger model"
| you'd have to email them.
|
| I guess, the "open" part of it is mostly for marketing.
| seydor wrote:
| Well i can see it becoming sexy soon
| turnsout wrote:
| Soon? Is the model filtered/censored?
| jamilton wrote:
| From the readme:
|
| >Bark has the capability to fully clone voices - including
| tone, pitch, emotion and prosody. The model also attempts to
| preserve music, ambient noise, etc. from input audio.
| However, to mitigate misuse of this technology, we limit the
| audio history prompts to a limited set of Suno-provided,
| fully synthetic options to choose from for each language.
|
| It's not immediately clear how the audio history prompts are
| created.
| joshjob42 wrote:
| I don't know how they're made exactly, but one can just
| edit the code a bit and delete the restriction to just the
| given audio history prompts. It's literally just enforced,
| affect, with a simple "assert" command.
| gkucsko wrote:
| history prompts are just unconditionally generated TTS from
| the same model. any of those can be used as history, but
| for convenience 10 are provided for each language (to
| generate things with consistent voices)
| turnsout wrote:
| So the history prompts are collections of text/audio
| pairs?
| gkucsko wrote:
| history is semantic, coarse and fine. so essentially the
| same thing thats getting generated just using it as an
| input before the generation
| xingped wrote:
| Does it sound fairly robotic/static-y to anyone else or just me?
| Doesn't sound any better than any other TTS software I've tried
| and in fact sounds a bit worse, like it's noisy.
| vlugorilla wrote:
| Great news! It's astounding how quickly technology is advancing.
| Only yesterday, I was wondering about when a new model for text-
| to-speech would be developed, and today a game-changing model has
| been released! This new model is simply incredible!
| [deleted]
| treerunner wrote:
| Can someone briefly explain how a model for a specific language
| is made for this tool?
| bdg wrote:
| Ok, the German example caught me -- it's too real. "But maybe it
| would be faster if..."
| cyberax wrote:
| On the other hand, Russian was disappointing. It put a stress
| in one word incorrectly (it confused the grammatical form, used
| the genitive case instead of the accusative) and in general
| sounded strange.
| computerex wrote:
| How do I save the audio array to a file on the file system?
| wgreenberg24 wrote:
| Awesome stuff! Can't wait to see where this company goes
| ripperdoc wrote:
| Am I hallucinating or didn't several of the examples have
| background audio artifacts, like it's been trained on speech with
| noisy backgrounds, I'm guessing audio from movies paired with
| subtitles? Having random background audio can make it quite hard
| to use in production.
| CreepGin wrote:
| I hear it too. I don't know if it's just background noise
| though. May be quality issues with the audio synthesis.
| gkucsko wrote:
| yeah sometimes there are definitely artifacts. technically
| they can be removed pretty easily with another model (like
| denoiser from FB) but for now we wanted to keep it simple to
| learn to control these things better through prompt
| engineering. Like when using a high quality input prompt it
| generally continues with high quality
| meepmorp wrote:
| At least in the last example, with the man and woman and
| the expensive oat milk, the background noise seemed to fit
| a likely public conversation scenario. I wasn't sure if it
| was accidental or not.
| JonathanFly wrote:
| >Am I hallucinating or didn't several of the examples have
| background audio artifacts, like it's been trained on speech
| with noisy backgrounds, I'm guessing audio from movies paired
| with subtitles? Having random background audio can make it
| quite hard to use in production.
|
| The other side of that problem is an opportunity. That's why
| the same model can also generate music, background noise and
| sound effects. And it's just because the prompt specifies those
| things explicitly. The input is truly semantic, so the output
| is rich and reflects that context. Is your input text sounds
| like it came from a speech, then there's a high chance your
| output audio will sound like a megaphone in a public space with
| crowd reactions and maybe even applause.
| nathias wrote:
| I hope you reconsider the misuse mitigation, I'm trying to clone
| my voice, but so far the other tools weren't that great ...
| newswasboring wrote:
| The misuse mitigation is 3 assert statements in one file. Its
| pretty easy to comment out.
| generalizations wrote:
| Is there likely to be a way to stream this audio in the future?
| As in, here's an incoming stream of text, generate the audio on-
| the-fly instead of all at once.
| quaintdev wrote:
| Good naming choice after OpenAIs whisper for speech to text. Bark
| is fitting name for text to speech.
| gkucsko wrote:
| Hey, one of the Suno founders/creators of Bark here. Thanks for
| all the comments, we love seeing how we can improve things in the
| future. At Suno we work on audio foundation models, creating
| speech, music, sounds effects etc....
|
| Text to speech was a natural playground for us to share with the
| community and get some feedback. Given that this model is a full
| GPT model, the text input is merely a guidance and the model can
| technically create any audio from scratch even without input
| text, aka hallucinations or audio continuation.
|
| When used as a TTS model, it's very different from the awesome
| high quality TTS models already available. It produces a wider
| range of audio - that could be a high quality studio recording of
| an actor or the same text leading to two people shouting in an
| argument at a noisy bar. Excited to see what the community can
| build and what we can learn for future products.
|
| Please let us know with any feedback, or if you're interested in
| working on this: bark@suno.ai
| ttul wrote:
| This tech will be used by crooks to automate attacks. Generate
| the language using GPT-4 and the audio using Bark, and then
| start making phone calls. Because it's open source, all you
| need is GPUs. This is not a criticism. I'm impressed and
| grateful for the openness. Everyone needs to wake up and
| recognize that these attacks are coming at us essentially right
| now.
| dmix wrote:
| How are the voices determined? Is there an option or is it just
| random/based on the prompts like "WOMAN"?
| turnsout wrote:
| Amazing work so far! Do you have any sense about how difficult
| it would be to enable M1/M2 or CoreML support?
| gkucsko wrote:
| thanks, the model itself is a pretty vanilla gpt model based
| heavily on karpathy's nanogpt, so should not need too many
| bells and whistles to get it running on specific
| architectures. that said i have very little experience with
| platform specific development, so would looove some help from
| the community :)
| ttul wrote:
| A serious nod to Karpathy here. They could have chosen any
| other Transformer architecture, but chose perhaps the most
| reachable one - in the literal sense.
| tmzt wrote:
| Would the same apply to a GGML port or are the architechtures
| too different?
| [deleted]
| causi wrote:
| Some of it is very impressive although some of it seems about
| equal to the TTS built into my phone. How long until someone can
| package this up and make a program that takes in epubs and spits
| out mp3s?
| billconan wrote:
| I tried it. It seems to hallucinate easily. The generated audio
| isn't what I provided.
|
| It seems to be easily reproducible if I specify a non-existing
| speaker?
|
| audio_array = generate_audio(text_prompt, 'en_speaker_3')
___________________________________________________________________
(page generated 2023-04-20 23:01 UTC)