[HN Gopher] QuiLLMan: Voice chat with Vicuna-13B
___________________________________________________________________
QuiLLMan: Voice chat with Vicuna-13B
Author : pierremenard
Score : 151 points
Date : 2023-04-25 15:09 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| yewenjie wrote:
| Does anybody know if there is an easy-to-deploy setup for
| tortoise-tts only?
| pierremenard wrote:
| Yup, the code in the above repo is based on
| https://github.com/metavoicexyz/tortoise-tts-modal-api
| teacpde wrote:
| Why is it serverless? It clearly has an API server.
| pierremenard wrote:
| The "API server" only runs when requests come in. Otherwise
| Modal scales everything down to 0, and you don't pay for
| anything. It's similar to putting Lambda behind an API gateway.
| SparkyMcUnicorn wrote:
| Because the resources are on demand, and the provider takes
| care of all the server management.
|
| https://en.wikipedia.org/wiki/Serverless_computing
| b33j0r wrote:
| I think serverless just means that you hire less devops people
| for your specific team.
|
| It's a cost-saving measure that costs you mortal wounds in
| integration expense, and eventually building out a real
| infrastructure plan.
| ZephyrBlu wrote:
| Is this the first time you've heard the term serverless? It
| means your code is plug & play, you don't have to manage the
| infra (I.e. a server) yourself.
| andybak wrote:
| I know the term serverless - but it was fairly ambiguous in
| this context.
|
| I presumed the same as the person you're replying to.
| teacpde wrote:
| I understand the concept of serverless as in AWS Lambda, but
| my first impression reading the title is serverless as in no
| backend. That would be a lot more amazing.
| explorigin wrote:
| The word you're looking for is "unhosted"
| esafak wrote:
| It _should_ be serverless but Amazon ruined it.
|
| As Michael Bolton said in Office Space: "Why should I
| change my name? He's the one who sucks."
| yieldcrv wrote:
| "serverless" is a throwback to 2013 a decade ago when it was a
| novelty to not know where your servers were or what they were
| made of, some people never move on. but in this case, this is a
| company advertising their serverless architecture for that
| specific crowd.
|
| this is distinct from "client side" where no servers are
| involved, aside from maybe something that serves a static web
| page.
| loudmax wrote:
| Sorry if this is off-topic, but those are some really good book
| recommendations in the demonstration image! If those are coming
| from Vicuna, that speaks well of it.
| tasty_freeze wrote:
| I pitched this on a recently thread, but it was 12+ hours after
| it was posted, so I'll try again here.
|
| What I really want is a program to waste the time of phone calls
| making unsolicited sales pitches.
|
| It would do voice to text, run a simple language model to
| generate responses, then synthesize the voice back. It doesn't
| need to be a sophisticated model, not much more sophisticated
| than the classic "Eliza" program. A few years back someone did
| this with a canned loop of vague responses and it fooled the
| sales people for surprisingly long:
|
| https://www.youtube.com/watch?v=XSoOrlh5i1k
|
| It seems like it could all run locally for low latency. Probably
| the most important part to get right would be a TTS system that
| isn't immediately pegged as a robot.
| Zetaphor wrote:
| Have you seen Lenny? It's a much lower tech solution that seems
| to work quite well.
|
| https://lennytroll.com/
| tasty_freeze wrote:
| Yes, that was the youtube video I linked to. If that can be
| successful with some telemarketers, I'm sure a chat-gpt aided
| program could be even more successful at frustrating
| telemarketers.
| [deleted]
| mdaniel wrote:
| just in case someone finds that YT video entertaining, you'll
| really love https://www.youtube.com/@KitbogaShow
| [deleted]
| lxe wrote:
| For "actually serverless" voice chat, check out
| https://whisper.ggerganov.com/
| antman wrote:
| Which of the models did you use?
| lxe wrote:
| This isn't me, this is ggerganov's work. His demo is using
| gpt2, and whisper models (https://ggml.ggerganov.com/).
| jancsika wrote:
| This link is the most worthwhile hijack of an HN thread that
| I've ever seen.
|
| How is tiny.en so damned accurate?
|
| How much faster does this run natively?
|
| Do the NEON instructions work for arm devices like an RPI or is
| it just tuned for Apple?
|
| With this and Alpaca 13B you could probably replace an entire
| window manager.
|
| Edit: it seems like it was less than a year ago that local
| speech recognition was a slow slog through a swamp of crufty,
| complicated research projects with most the high quality
| training data hidden behind walled gardens. Now I can stutter-
| step over a word and a demo in my browser correctly transcribes
| what I meant sans stutter. What happened?
| anotherhue wrote:
| Yes NEON works on recent Pis. RPi CPU tuning: https://github.
| com/ggerganov/whisper.cpp/blob/70567eff232773...
| lxe wrote:
| SIMD support in wasm is a bit weird. You'll need to compile
| into multiple targets, and many compile flags are not
| supported: https://emscripten.org/docs/porting/simd.html
| teacpde wrote:
| Really cool, I was able to play with it for a bit, but now
| seems like HN hug-of-death kicked in.
| lxe wrote:
| It's hard to death-hug these wasm demos, as they are all
| static files that can be easily served via CDNs
| EGreg wrote:
| what is the point when the Web browsers mostly all already
| support it
|
| https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_...
| lxe wrote:
| Engineering curiosity and fun
| easrng wrote:
| In Chrome that's using Google's servers to transcribe the
| audio (last I checked) and it doesn't work on Firefox.
| hombre_fatal wrote:
| When you want much better, state of the art quality.
|
| Compare it yourself.
| EGreg wrote:
| The browser seems to do very well for me, and in multiple
| languages. What about this, can it do languages?
| hombre_fatal wrote:
| Whisper can do you one better: you don't need to know the
| spoken language ahead of time, and you can mix languages
| in the same sentence.
|
| It does all sorts of things better like upgrading words
| into proper nouns, and doing it based on context.
|
| Honestly it's the first time speech recognition finally
| feels "production ready" after years of Siri/Alexa.
| RockRobotRock wrote:
| Whisper supports many different languages.
| arthurcolle wrote:
| This is so good. I tried playing around with whisper.cpp last
| week and got absolutely terrible performance. I played around
| with the "step size" as well as "n" (not sure what these do, I
| should probably read the docs seriously).
|
| I tried a lot of tweaks but this one is definitely the best
| I've seen.
|
| How did it know how to spell my name correctly when I just
| spoke into my microphone??? The two L's usually trips up the
| transcription models. What????
| dom96 wrote:
| wow, you weren't kidding. I tried the tiny model and it got
| what I said perfectly. Super impressive.
| syntaxing wrote:
| Hmm curious what the differences are, it's the same person
| who wrote whisper cpp that hosts that website.
| kurisufag wrote:
| >How did it know how to spell my name correctly when I just
| spoke into my microphone???
|
| if the variations are pronounced the same? luck, probably.
| arthurcolle wrote:
| My last name is pronounced "Colle" but when I've tried
| other whisper demos it always spelled it Cole or Coll.
|
| This is the first time I've ever seen it get the spelling
| actually right (missing the accent is forgivable) ;)
| breakingrules wrote:
| [dead]
| IronWolve wrote:
| Been using textgen and downloading tons of models, the models are
| all over the place. The problems of accuracy and short term
| memory are major issues that people are trying to implement work
| arounds.
|
| Check out textgen, it has voice in/out, graphics in/out, memory
| plugin, api, plugins, etc, all running locally.
|
| https://github.com/oobabooga/text-generation-webui
| bioemerl wrote:
| > it has voice in/out,
|
| Do you know how to get this working? I looked through the read-
| me and didn't see any options for it.
| IronWolve wrote:
| https://github.com/oobabooga/text-generation-
| webui/blob/main...
|
| You need to enable the extensions.
|
| I only did voice out locally with silero_tts, it also
| supports voice out with eleven labs api.
|
| Voice input is via whisper tts.
| CrazyStat wrote:
| Any models and/or plugins you recommend?
| wongarsu wrote:
| That's a pretty cool showcase of modal [1]. From a marketing
| perspective I have to congratulate, this is a really well done
| way to get people to check out your platform.
|
| 1: https://modal.com/
| turnsout wrote:
| Is it basically Vercel for ML?
| fredliu wrote:
| +1 modal.com is the first thing I checked after reading the
| readme.
| bredren wrote:
| I also clicked through but immediately abandoned wondering:
|
| What's the state of container based ML deployments?
|
| Can I take a container orchestration if these services and just
| put them on a vps w a GPU and run this?
|
| Is there secret or just special sauce in ML infra?
| npace12 wrote:
| +1, props to modal.com, well done and their site is nice
| forgingahead wrote:
| Nice to see Tortoise being used - I still think it's the best TTS
| system out there now. Generation time is slow, but quality is
| incredible. I wonder if the code can be optimised to speed up the
| generation, but I don't think the author is maintaining it any
| longer.[0]
|
| [0]https://github.com/neonbjb/tortoise-tts
| Tepix wrote:
| Coqui also looks interesting.
|
| https://github.com/coqui-ai/TTS
|
| Support for it was recently added to vocode:
|
| https://github.com/vocodedev/vocode-python/pull/56
| depingus wrote:
| You might want to check out Mimic 3 by the Mycroft team. Its
| also open source and runs offline.
|
| https://mycroft.ai/mimic-3/
| bondsynth wrote:
| If you're generally interested in TTS check out Bondsynth AI,
| it's a product I've been working on for long-form text-to-
| speech (think ebook to audiobook or website to audio). It's
| still in beta, so it's free, but I'm looking for feedback.
|
| Demo at: https://www.youtube.com/watch?v=OmQup3kst5s
|
| Signup at: https://signup.bondsynth.ai
| dontwearitout wrote:
| Very impressive demo! FYI https://signup.bondsynth.ai doesn't
| work, but https://bondsynth.ai/signup.html does
| generalizations wrote:
| There's a lot of use cases just waiting for a good system that
| can do at least live speed generation.
| turnsout wrote:
| Tortoise looks really nice! The output is very "polished" and
| audiobook-like. It's a contrast to Bark[0] which is far more
| expressive but unpredictable.
|
| [0]: https://github.com/suno-ai/bark
| lee101 wrote:
| [dead]
| syntaxing wrote:
| Off tangent but can someone at Apple please just replace the Siri
| word recognition with whisper. We can finally have multi language
| support and not dogshit recognition.
| juliennakache wrote:
| Interesting. How does local development work or remote debugging
| work if the entire production toolchain is abstracted away with
| proprietary software?
| thundergolfer wrote:
| This docs guide has some answers:
| https://modal.com/docs/guide/developing-debugging
| sramam wrote:
| Very cool - the demo was simple, functional and clear.
|
| It was a bit laggy, but for a free demo from an open source
| project, I should be the one being shamed!
|
| Well done.
___________________________________________________________________
(page generated 2023-04-25 23:01 UTC)