[HN Gopher] Sopro TTS: A 169M model with zero-shot voice cloning...
___________________________________________________________________
Sopro TTS: A 169M model with zero-shot voice cloning that runs on
the CPU
Author : sammyyyyyyy
Score : 66 points
Date : 2026-01-08 20:37 UTC (2 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| lukebechtel wrote:
| Very cool. I'd love a slightly larger version with hopefully
| improved voice quality.
|
| Nice work!
| sammyyyyyyy wrote:
| Thanks! Yeah I kinda postponed publishing it until it was a bit
| better, but as a perfectionist, it would have never been
| published
| convivialdingo wrote:
| Impressive! The cloning and voice affect is great. Has a slight
| warble in the voice on long vowels, but not a huge issue. I'll
| definitely check it out - we could use voice generation for
| alerting on one of our projects (no GPUs on hardware).
| sammyyyyyyy wrote:
| Cool! Yeah the voice quality really depends on the reference
| audio. Also mess with the parameters. All the feedback is
| welcome
| realityfactchex wrote:
| That's cool and useful.
|
| IMO, the best alternative is Chatterbox-TTS-Server [0] (slower,
| but quite high quality).
|
| [0] https://github.com/devnen/Chatterbox-TTS-Server
| blitzar wrote:
| Mission impossible cloning skills without the long compile time.
|
| "The pleasure of Buzby's company is what I most enjoy. He put a
| tack on Miss Yancy's chair ..."
|
| https://www.youtube.com/watch?v=H2kIN9PgvNo
|
| https://literalminded.wordpress.com/2006/05/05/a-panphonic-p...
| btbuildem wrote:
| It's impressive given the constraints!
|
| Would you consider releasing a more capable version that renders
| with fewer artifacts (and maybe requires a bit more processing
| power)?
|
| Chatterbox is my go-to, this could be a nice alternative were it
| capable of high-fidelity results!
| sammyyyyyyy wrote:
| This is my side "hobby". And compute is quite expensive. But if
| the community's responsive is good, I will definitely think
| about it! Btw, chatterbox is a great model and inspiration
| elaus wrote:
| Very nice to have done this by yourself, locally.
|
| I wish there was an open/local tts model with voice cloning as
| good as 11l (for non-english languages even)
| sammyyyyyyy wrote:
| Yeah, we are not quite there, but I'm sure we are not far
| either
| SoftTalker wrote:
| What does "zero-shot" mean in this context?
| nateb2022 wrote:
| > Zero-shot learning (ZSL) is a problem setup in deep learning
| where, at test time, a learner observes samples from classes
| which were not observed during training, and needs to predict
| the class that they belong to. The name is a play on words
| based on the earlier concept of one-shot learning, in which
| classification can be learned from only one, or a few,
| examples.
|
| https://en.wikipedia.org/wiki/Zero-shot_learning
| woodson wrote:
| This generic answer from Wikipedia is not very helpful in
| this context. Zero-shot voice cloning in TTS usually means
| that data of the target speaker you want the generated speech
| to sound like does not need to be included in the training
| data used to train the TTS models. In other words, you can
| provide an audio sample of the target speaker together with
| the text to be spoken to generate the audio that sounds like
| it was spoken by that speaker.
| coder543 wrote:
| Why wouldn't that be one-shot voice cloning? The concept of
| calling it zero shot doesn't really make sense to me.
| geocar wrote:
| So if you get your target to record (say) 1 hour of
| audio, that's a one-shot.
|
| If you didn't do that (because you have 100 hours of
| other people talking), that's zero-shots, no?
| woodson wrote:
| I don't disagree, but that's what people started calling
| it. Zero-shot doesn't make sense anyway, as how would the
| model know what voice it should sound like (unless it's a
| celebrity voice or similar included in the training data
| where it's enough to specify a name).
| ben_w wrote:
| Zero-shot means zero-retraining, so think along the lines
| of "Do you need to modify the weights? Or can you keep
| the weights fixed and you only need to supply an
| example?"
|
| As with other replies, yes this is a silly name.
| derefr wrote:
| Is there yet any model like this, but which works as a "speech
| plus speech to speech" voice modulator -- i.e. taking a fixed
| audio sample (the prompt), plus a continuous audio stream (the
| input), and transforming any speech component of the input to
| have the tone and timbre of the voice in the prompt, resulting in
| a continuous audio output stream? (Ideally, while passing through
| non-speech parts of the input audio stream; but those could also
| be handled other ways, with traditional source separation
| techniques, microphone arrays, etc.)
|
| Though I suppose, for the use-case I'm thinking of (v-tubers),
| you don't really need the ability to dynamically change the
| prompt; so you could also simplify this to a continuous single-
| stream "speech to speech" model, which gets its target vocal
| timbre burned into it during an expensive (but one-time) fine-
| tuning step.
| vunderba wrote:
| I don't know about open models, but ElevenLabs has had this
| idea of mapping intonation/emotion/inflections onto a
| designated TTS voice for a while.
|
| https://elevenlabs.io/blog/speech-to-speech
| gcr wrote:
| Chatterbox TTS does this in "voice cloning" mode but you have
| to implement the streaming part yourself.
|
| There are two inputs: audio A ("style") and B ("content"). The
| timbre is taken from A, and the content, pronunciation,
| prosody, accent, etc is taken from B.
|
| Strictly soeaking, voice cloning models like this and
| chatterbox are not "TTS" - they're better thought of as
| "S+STS", that is, speech+style to speech
| qingcharles wrote:
| There must be something out there that does this reliably as I
| often see/hear v-tubers doing it.
| nunobrito wrote:
| Muito fixe. Now the next challenge (for me) is how to convert
| this to DART and run on Android. :-)
| sammyyyyyyy wrote:
| Obrigado! Quando (e se fizeres isso) manda pm!
| woodson wrote:
| Does the 169M include the ~90M params for the Mimi codec?
| Interesting approach using FiLM for speaker conditioning.
| sammyyyyyyy wrote:
| No, it doesn't.
| brikym wrote:
| A scammers dream.
| jacquesm wrote:
| That's exactly how I see it.
| jacquesm wrote:
| What could possibly go wrong...
|
| Don't you ever think about what the balance of good and bad is
| when you make something like this? What's the upside? What's the
| downside?
|
| In this particular case I can only see downsides, if there are
| upsides I'd love to hear about them. All I see is my elderly
| family members getting 'me' on their phones asking for help, and
| falling for it.
|
| I've gotten into the habit of waiting for the other person to
| speak first when I answer the phone now and the number is unknown
| to me.
| sammyyyyyyy wrote:
| Yes, you are right. However, there are many upsides to this
| kind of technology. For example, it can restore the voices of
| people who were affected by numerous diseases
___________________________________________________________________
(page generated 2026-01-08 23:00 UTC)