Self-hosted text-to-speech and voice cloning - review of Coqui reddit / selfhosted ---------------------------------------------------------------------- opensourcecolumbus · 2023-11-05 11:58 Have been researching about Open Source tools for converting text-to- speech. And until recently, it seemed like there's no practically decent solution which is free and easy to self host. **Coqui TTS** started looking like a decent solution a month ago, since then I have beem using it and I have a mixed feeling about. Here's the summary of the review for Coqui TTS. [Originally poated on #OpenSourceDiscovery n ewsletter](https://opensourcedisc.substack.com/p/opensourcediscovery- 83-coqui-tts/) Project: **Coqui TTS (A deep learning toolkit for Text-to-Speech)** > Clone voices and generate speech from text with pertained models in +1100 languages * Demo : [Cloned voice of steve jobs](https://opensourcedisc.substack. com/p/opensourcediscovery-83-coqui-tts) * Source: https://github.com/coqui-ai/tts * Stack: Python * Author: Eren Gölge and Coqui team * License: MPL 2.0 💖 What's good about Coqui: * Quick and lightweight installation * Decent text-to-speech output * Supports multiple TTS models and fine-tuning methods 👎 What can be improved: * Cloned voice does not feel like clone (although it did had some features of the source voice) * Underlying XTTS model is not open-source ⭐ Ratings and metrics * Production readiness: 7/10 * Docs rating: 7/10 * Time to POC(proof of concept): more than a week Note: This is a summary of the full review posted on [#OpenSourceDiscovery](https://opensourcedisc.substack.com) newsletter. I have more thoughts on each points and would love to answer them in comments. Would love to hear your experience ---------------------------------------------------------------------- opensourcecolumbus · 2023-11-05 12:00 Keen to hear your experience, specially if you had success in finetuning cloned voice to match the source ---------------------------------------------------------------------- Plain-Tangerine3715 · 2023-11-05 15:36 I've only just dipped my toe in this space, but I'm also very interested in what's possible with a self hosted and open source solution for voice cloned tts. I was using tortoise tts: [https://github.com/neonbjb/tortoise- tts](https://github.com/neonbjb/tortoise-tts) quick observation * the docker setup was mostly painless, but there was a tweak to the supplied docker file that must be made to get it to run (documented in the issues on git hub) * looks like in this space they expect your to have an nvidia gfx card, I do not and while it did still work out of the box, it was pretty slow, which I guess is expected. It's my understanding tts with tortoise is much faster with a device. There were folks that got tortoise to be accelerated with radeon cards, but I have not tried to reproduce that but that's next. * The results with "ultra-fast" preset were decent, I have high hopes for "high-quality" preset, but I will first try to get the process accelerated on my radeon. * I was generating my first samples in less than 2 hours. I have not tried to do a clone with this yet. Why did you choose coqui to check out and were there others you considered? ---------------------------------------------------------------------- tehnomad · 2023-11-05 17:42 From what I've tried, [RVC](https://github.com/ddPn08/rvc-webui) seems to be the best at cloning voices. ---------------------------------------------------------------------- lilolalu · 2023-11-05 18:37 Did you actually try to clone your voice? For me, none of them worked. ---------------------------------------------------------------------- tehnomad · 2023-11-05 19:00 I tried using a 3 min. sample of me speaking that didn't come out great, but I think I need more training data. But I've seen pretty impressive results on Youtube. ---------------------------------------------------------------------- lilolalu · 2023-11-05 19:01 i tried with much longer and much shorter samples, didnt work. also the feedback on the github doesnt sound that tghe voice cloning actually works right now. ---------------------------------------------------------------------- digitalindependent · 2023-11-05 20:05 Really like that you have started adding „What I like/dislike“. That makes it really interesting to read and learn from your experiences. Subscribed! ---------------------------------------------------------------------- opensourcecolumbus · 2023-11-06 02:30 Thank you for sharing the feedback. Really helps me make these posts more and more useful. ---------------------------------------------------------------------- opensourcecolumbus · 2023-11-06 02:46 Thank you for sharing. Glad you asked. My primary focus was on quality over speed/training-time. The benchmark I had was eleven labs output. Top 3 tools I found were * Coqui TTS * Mozilla TTS (ruled out because coqui is the successor of this one) * Tortoise (HF space demo didn't work, it seemed to have some runtime, docs were not as good as coqui, coqui seemed to be more active in resolving issues than tortoise) All of this led me to delay experimenting with tortoise. Now that I have tried Coqui, I'm not sure what is it that Tortoise does differently that can result in better outcome. Might invest time in trying tortoise as well if I have clear answer to that. Should I? ---------------------------------------------------------------------- opensourcecolumbus · 2023-11-06 02:54 Can you please share the English docs, I'm finding it hard to understand this project. ---------------------------------------------------------------------- tehnomad · 2023-11-06 19:24 This is the most recent WebUI fork that I used: https://github.com/IAHispano/Applio-RVC-Fork ---------------------------------------------------------------------- CheatCodesOfLife · 2023-12-22 05:25 It worked fine for me. I used it on people without telling them it's my voice, and was always told "Hey that sounds like you!" I read this out: "The examination and testimony of the experts; enabled the commision to conclude; that 5 shots may have been fired." Export it as a mono .wav file, 22050hz. ---------------------------------------------------------------------- lilolalu · 2023-12-22 09:52 Yeah, the generation quality is one issue, the actual sound quality another. I have been "repairing" generated TTS samples with "vocos" which worked quite well. ---------------------------------------------------------------------- YLSP · 2023-12-24 02:53 Did you try the mrq version of Tortoise TTS. Unfortunately the author was quite active up until mid-November. I suspect either (A) something horrible happened to the author, or (B) someone hired the author based on his work with this tool and his terms of hiring were that he could no longer contribute. Maybe even 11Labs paid him to not contribute to his project anymore. https://git.ecker.tech/mrq/ai-voice-cloning The difference between this and Tortoise is that the original author of TortoiseTTS did not make some of the cloning features available. I have found that It is a very good tool to clone voices.... ---------------------------------------------------------------------- DashinTheFields · 2023-12-29 20:27 Do you know of a tool that does multi-track? I would like to provide it a story, like json format \[{name: value, text: value}\], but with multiple characters, and then have it output. Kind of like any studio software. ---------------------------------------------------------------------- opensourcecolumbus · 2023-12-30 17:07 If I remember correctly Coqui Studio does exactly that but I don't that was OSS. ---------------------------------------------------------------------- DashinTheFields · 2023-12-30 19:19 >Yeah, it's not self hosted. So it doesn't fit the criteria of /selfhosted. I found a person on here who says they're interested in the idea, who has done some development on other coqui projects. So I guess I'll repost if they do anything. ---------------------------------------------------------------------- badcookie911 · 2023-12-31 10:20 Personally I have tried Coqui TTS with their XTTS model, Tortoise and 11labs. In term of TTS, hands down 11labs is the best in quality, but when you start fiddling with voice cloning, a lot other factors in play. 11labs instant voice cloning is OK, the professional voice cloning requires user authentication, meaning you can't clone anyone without them doing the verification. And it takes 3 weeks. Coqui XTTS fine tuning works great in voice cloning, 7/10 if clone normal voice. I find it hard to clone gaming character voice and anime female voice with high pitch. TortoiseTTS is a good TTS, but it is slow, not suitable for conversational use. RVC is a speech to speech STS voice cloning. Quality is good but you need to have a good TTS source that ideally shares similar vocal range as our voice clone, because you have to first generate the voice with TTS, then convert to your target voice with RVC (STS). ---------------------------------------------------------------------- badcookie911 · 2023-12-31 10:21 If anyone manage to fine tune a good TTS for anime girl voice please do share :)