[HN Gopher] Ask HN: What do you use for ML Hosting?
___________________________________________________________________
Ask HN: What do you use for ML Hosting?
I'm trying to setup server to run ML inferences. I need to
provision a somewhat beefy gpu with a decent amount of RAM (8-16
GB). Does anyone here have personal experience and recommendations
about the various companies operating in this space?
Author : blululu
Score : 88 points
Date : 2023-05-02 18:58 UTC (4 hours ago)
| bfirsh wrote:
| Founder of https://replicate.com/ here, which has been mentioned
| a few times. Happy to help you get set up. :) ben@replicate.com
| zitterbewegung wrote:
| Have you tried self hosting? All you need is business internet
| with a static IP which is quite inexpensive and doing inference
| can be done on CPU depending what you want to perform inferfence.
| Also, wherever you are hosting a good rule of thumb is to have at
| least 1.5 times the amount of regular ram with your VRAM.
| languagehacker wrote:
| Vultr GPU: https://www.vultr.com/products/cloud-gpu/
| sa-code wrote:
| What about just using a cloud VM with an ansible script? I find
| ML deployment solutions to be very over engineered
| edunteman wrote:
| Hey! Would love to have you try https://banana.dev (bias: I'm one
| of the founders). We run A100s for you and scale 0->1->n->0 on
| demand, so you only pay for what you use.
|
| I'm at erik@banana.dev if you want any help with it :)
| aaronharnly wrote:
| Looks great. It appears aimed at the inference use case rather
| than training, yes?
| edunteman wrote:
| yeah, we're optimizing the infra for realtime inference,
| though people definitely still do run training on us, with a
| weights upload implemented at the end of your handler.
| jaflo wrote:
| +1 on banana.dev, I used it for a side project and deployed
| some custom code and it was a good experience! I liked the
| pricing model (lack of minimums and pay for usage instead of a
| "plan") and how you can package up whatever code you want.
| samstave wrote:
| Kind of funny how cellphones went the opposite way - we all
| hated "paying for usage (minutes/txts))" and now we want just
| the Plan.
| alpaca128 wrote:
| I still use that because I rarely use the phone without
| wifi and don't make a lot of calls. And because it's 2023 I
| can change to a paid plan for a month at any time in the
| app, so it's the best of both worlds.
| edunteman wrote:
| Thanks for the +1!
|
| Small note here: our billing is changing within the next
| month, to up-front payments that apply as a credit balance to
| your account. It still won't have minimums and you'll have
| the option to set up auto-refill on your balance, so it will
| functionally remain pay-as-you-go, but just wanted to add
| flavor to your comment on the pricing model.
|
| Thanks for using us btw, you rock
| senko wrote:
| Looking at your pricing, is that per seconds of GPU usage or
| per total seconds the app is running?
|
| Eg. might have only a few minutes of usage in an hour and the
| rest of the time is spent waiting for requests. How's that
| billed?
| edunteman wrote:
| https://docs.banana.dev/banana-docs/core-concepts/billing
| You're only billed for active replica time. Call comes in, we
| start a replica, it handles the request, it waits around for
| a 10s (configurable) idle timeout to handle any additional
| calls, and shuts down if no other calls to serve. The idle
| timeout is to prevent cold boots when not necessary, but is
| billed, so you can get closer to pure pay-per-call pricing by
| reducing idle timeout.
| asadm wrote:
| I have had good experience with Replicate and Runpod. Replicate
| seems to be nicer but has very bad cold boot issue. Runpod is
| great once you have an app set up!
|
| I use mix of both for my side project: https://trainengine.ai
| bfirsh wrote:
| Founder of Replicate here. Looks like you're using DreamBooth
| for TrainEngine. We have a beta version of really fast cold
| boots for DreamBooth trainings. I'll drop you an email to get
| you set up with it.
|
| We've got some big cold improvements rolling out across
| everything soon. We can also just keep models switched on to
| avoid cold boots entirely.
| chaoyu_ wrote:
| Check out BentoML https://github.com/bentoml
| elforce002 wrote:
| +1 for BentoML. Open source, good docs, and the community
| around it is responsive.
| _boffin_ wrote:
| I'm using a docker container on Ubuntu, which is on my home lab
| that's an esxi 6.5 hypervisor. Going to be building a new machine
| with a few hundred GB of ram and then, at some point in the next
| 6 months, looking at getting a good GPU with a bunch of vRAM.
|
| Wrapped the thing in a flask app so I can expose APIs I build
| out.
| sjkoelle wrote:
| if you want to host voice ML models, check out Uberduck.
| tehsauce wrote:
| Vast.ai Nobody has better prices.
| mcsniff wrote:
| Another vote for vast.ai, has been around quite a while and
| I've been using them for shell access to bare metal machines
| stuffed with GPUs, always had a decent experience.
| Areibman wrote:
| Baseten was by far the easiest setup I've tried
| https://www.baseten.co
| efxhoy wrote:
| We use Sagemaker at work because AWS. I don't really like their
| style of APIs but it works.
| thundergolfer wrote:
| On Modal.com these 34 lines of code is all you need to
| serverlessly run BERT text generation inference on an A10G (which
| has 24GB of GPU memory). No Dockerfile, no YAML, no Terraform or
| AWS Cloudformation. Just these 34 lines. import
| modal def download_model(): from
| transformers import pipeline pipeline("fill-mask",
| model="bert-base-uncased") CACHE_PATH =
| "/root/model_cache" # model location in image ENV =
| modal.Secret({"TRANSFORMERS_CACHE": CACHE_PATH}) image
| = ( modal.Image.debian_slim()
| .pip_install("torch", "transformers")
| .run_function(download_model, secret=ENV) ) stub =
| modal.Stub(name="hn-demo", image=image) class
| Model: def __enter__(self): from
| transformers import pipeline self.model =
| pipeline("fill-mask", model="bert-base-uncased", device=0)
| @stub.function( gpu="a10g",
| secret=ENV, ) def handler(self, prompt: str):
| return self.model(prompt) if __name__ ==
| "__main__": with stub.run(): prompt =
| "Hello World! I am a [MASK] machine learning model."
| print(Model().handler.call(prompt)[0]["sequence"])
|
| Running `python hn_demo.py` prints "Hello World! I am a simple
| machine learning model."
|
| You can check out available GPUs at
| https://modal.com/docs/reference/modal.gpu.
|
| There's also a bunch of easy-to-run examples in our docs :)
| https://modal.com/docs/guide/ex/stable_diffusion_cli
| codeptualize wrote:
| Love Modal. We use it for data processing, queues, apis, and
| all sorts of random things. Such a great product!
| therealmarv wrote:
| btw. HN supports very simple code formatting, just indent by
| two or more spaces https://news.ycombinator.com/formatdoc
| thundergolfer wrote:
| Ah nice. Thank you. I was using backticks
| outdoorblake wrote:
| Banana.dev is what I use. The cold boots are fast
| howon92 wrote:
| Here are some candidates: - HuggingFace Inference Endpoints:
| https://huggingface.co/inference-endpoints - Amazon SageMaker:
| https://aws.amazon.com/sagemaker/ - Replicate:
| https://replicate.com/
|
| The first two are more customizable than the last. SageMaker is
| the cheapest.
| version_five wrote:
| My preference is not to have to change my code to use some
| special framework, and just get access to a gpu machine I can run
| my stuff on.
|
| I'm assuming you know what you need for a GPU. If you're unsure,
| consider trying to run inferences on a CPU and see how long it
| takes and if it could work.
|
| And then just look at price and reliability for a gpu machine
| with the different cloud providers. Ovh is cheap but the only
| thing worse than their reliability is their customer service.
| Various niche players offering V100s used to pop up that were
| pretty cheap. AWS is more expensive, more reliable, they may
| still have availability problems. Paperspace looks pretty good.
| Etc.
| thundergolfer wrote:
| > are worth avoiding so you don't get stuck with somebody
| else's framework.
|
| Modal eng here. Modal is not setup as a framework. Think of
| more as Python-defined serverless infrastructure that has
| native support for the Python runtime. This is in some places
| called "Infrastructure from code", as opposed to
| "Infrastructure as code" which means just source-controlling
| K8s YAML and Cloudformation.
|
| A major benefit of this approach is that the cloud becomes part
| of your dev loop, as opposed to doing `docker build`, `docker
| push`, `kubectl`, etc just to ship a change to a GPU.
|
| In the script I posted Modal APIs are mixed in with standard
| Python code for brevity, but many customers just keep their
| code in their own modules and have a `modal_infra.py` module
| that defines the serverless infrastructure.
| version_five wrote:
| Understood, thanks for clarifying. I'll edit my post.
| naderkhalil wrote:
| That makes sense, Brev.dev is a really simple way to run your
| code on a configured GPU without having to change your code.
| It'll also optimize your GPU to save money when possible.
| jetml wrote:
| Check out JetML.com (I'm the founder). Happy to help get you
| started with a demo if you want to reach out nick@jetml.com.
| psshank wrote:
| Try www.salad.com. We've got 10k+ GPUs - from 8GB to 24GB. You
| get 10x more inferences per dollar compared to others. Our
| product team is pretty happy to help out on Discord. Some prices
| of interest. RTX 3060 - 12 GB - $0.08/hr RTX 3090 - 24 GB -
| $0.25/hr
| pj_mukh wrote:
| If you're using python, Modal (modal.com) was awesome to setup.
|
| They'll take a FastAPI setup too and just put it online to be
| used on demand.
| lee101 wrote:
| [dead]
| jvanillaaaa wrote:
| Brev.dev
|
| This is exactly what you're looking for
| lordofgibbons wrote:
| Do any of the "serverless"/saas model hosting services perform
| optimizations such as quantization or input micro-batching?
| tikkun wrote:
| For serverless: check the list I posted here
| https://news.ycombinator.com/item?id=34742087 (I ended up using
| Banana, it was fine)
|
| For non-serverless, some to check out are these (though likely
| all overkill if you just need a single GPU)
|
| https://www.coreweave.com/
|
| vast.ai
|
| Lambda labs
| thundergolfer wrote:
| How come you didn't end up using Modal, seeing at it was
| recommended in the only reply in the thread? [I'm a Modal
| person looking for insight :)]
| [deleted]
| smoldesu wrote:
| I'm currently running a Discord bot with a 7B model off a free
| Oracle Ampere instance with their Pytorch Accelerated[0] image.
| It's not terribly fast, but totally usable for group chats that
| want to interrogate an AI. If you're doing some sort of offline
| processing or non-time-imperative operation, something like this
| might be worth looking into.
|
| [0]
| https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...
| aditya wrote:
| what discord bot? :)
| password4321 wrote:
| Does that use all 4 OCPUs / 24GB memory?
| smoldesu wrote:
| It can! I'm using 2 cores per request though, and I've got
| memory to spare.
| bootsmann wrote:
| Oracles free forever tier is so underrated. They just throw
| half a startup at you at no cost.
| ihgautam wrote:
| KFServing
| rgbrgb wrote:
| Wow, looks like there's a ton of choices here I haven't looked
| at. For iterate.world we use replicate but just added kandinsky
| from runpod. Thinking about switching everything to runpod
| because it's 5-10x cheaper and we only use models that they have
| anyway.
|
| There's one I won't share that's is now defunct but you could use
| any diffuser's compatible project on Hugging Face, which was such
| a cool feature. I wish someone (cheap) would implement this!
|
| edit: just looked at banana.dev in this thread, their templates
| look closest to the HuggingFace integration though I don't think
| they have webhooks.
| edunteman wrote:
| Hey! Banana founder here. Explicit webhook support coming out
| soon, though one could always add an http POST request to their
| webhook endpoint at the end of their handler to send the data
| that way rather than awaiting the results from the client. It'd
| take some customization, but you're into our templates, you can
| click the github icon in the UI to see the source repo, fork
| it, add the HTTP POST call at the end of the handler, and then
| deploy that to Banana as a custom repo.
| bfirsh wrote:
| Founder of Replicate here. Also YC founder (W20). :)
|
| It's also worth noting that we bill by the second for how long
| your prediction is running, and we don't bill for any idle
| time, so in practice Replicate works out cheaper for many
| workloads. We can give discounts if you're putting through a
| decent amount of traffic. We should be able to match Runpod's
| pricing.
|
| Drop me an email: ben@replicate.com
___________________________________________________________________
(page generated 2023-05-02 23:01 UTC)