whisper self hosted what's the most cost-efficient way
reddit / selfhosted

----------------------------------------------------------------------
AlwaysBlue1 · 2023-10-16 00:56

i'm planing to to run a whisper on a self-hosted server can somebody
point me in the right direction to where I can run it with large 2
models at the cheapest and fastest way possible?

​

to give you some context we are doing large quantities of video to
text transcription for a SAAS product. whisper APi will be costly at
our scale also we don't need the 25 mb file size limit as our typical
file is around 150mb

​

​

----------------------------------------------------------------------
FalseRegister · 2023-10-16 01:16

Check whiper C/C++ implementation. It uses CPU only. If you don't need
real time but can analyze in a few minutes, then any server will do. I
did this for audio transcription before, worked better than other
solutions.

----------------------------------------------------------------------
Anposter · 2023-10-16 07:55

I've been using [Whisper ASR
Webservice](https://ahmetoner.github.io/whisper-asr-webservice) with
great success for shorter videos.
It's supports faster-whisper as well.

----------------------------------------------------------------------
xzaz · 2023-10-16 08:21

I thought you where talking about whisperr for a second.

----------------------------------------------------------------------
JiggySnoop · 2023-10-16 08:52

Lamo same.

----------------------------------------------------------------------
ozzeruk82 · 2023-10-16 09:32

It can actually be compiled with options to use graphics cards too, to
give a bit of a speedup.

I highly recommend it, I used it when at a trade show to quickly
transcribe a video on a Macbook Air and it did the job superbly in
very little time.

----------------------------------------------------------------------
FalseRegister · 2023-10-16 09:35

On any Apple Silicon it runs blazing fast! I agree

I didn't know they finally added support for GPU, that was not
available when I used it. Only on the mainstream implementation using
Python (IIRC).

----------------------------------------------------------------------
ozzeruk82 · 2023-10-16 09:38

Yup it’s a fairly recent addition I think

----------------------------------------------------------------------
lilolalu · 2023-10-16 11:31

There is a template to start Whisper ASR Webservice on vast.ai ... The
actual price will depend on the GPU you will rent for the purpose,
storage and bandwidth you need. GPU usually starts at around $0.150/hr
for a A4000 /w 16gb RAM and goes up to $0.5/hr for a rtx4090/24gb

I've been fiddling mainly with vast.ai recently as it seemed to be one
of the cheaper services, but there are a lot of GPU clouds nowadays,
salad.com etc.pp.

It's nice that you can fire up instances by demand, even
programmatically by cli. You can configure your own docker image to
run on the GPU hosts, a jupyter notebook (probably not the right
choice for your use case) or use one of their templates.

----------------------------------------------------------------------
trararawe · 2023-10-16 12:19

Checkout [whisperx](https://github.com/m-bain/whisperX)