[HN Gopher] The State of Serverless GPUs
___________________________________________________________________
The State of Serverless GPUs
Author : kiyanwang
Score : 58 points
Date : 2023-04-28 06:21 UTC (1 days ago)
(HTM) web link (www.inferless.com)
(TXT) w3m dump (www.inferless.com)
| chewbacha wrote:
| Serverless feels like a low-interest rate era technology.
| varelse wrote:
| [dead]
| blandcoffee wrote:
| Is this a sarcastic quip or are you able to expand on this?
|
| I use a lot of serverless daily, handling events (even ML
| inference), and it seems to work great, but would love to
| understand the alternatives and your perspective.
| chewbacha wrote:
| The overhead of abstracting away the servers is a luxury in
| many ways. This extra cost I believe was heavily funded by
| low-interest rates which flushed the VC world with dough.
| There's been a lot less serverless talk since the fed started
| cranking the rates
| itisit wrote:
| Sorry, but this feels like a total non sequitir. Serverless
| or FaaS is pretty mature now. People get the concept,
| businesses understand the savings, and the services and
| tooling are stable. We don't talk about it because it's
| boring.
| jeffybefffy519 wrote:
| Have you used serverless for anything in production before?
| dataangel wrote:
| Hard disagree, it's the opposite. Serverless helps increase
| utilization.
| ris wrote:
| Inferless provides Serverless GPU solutions
| touisteur wrote:
| Wondering whether most major inference libraries support storage-
| direct or if the listed providers are cheaping out on storage
| latency. Several seconds to load a 100MB model when pcie 4 is
| 256Gb/s I'd have expected an order of magnitude less - at least
| from my experience with gpudirect and real-time processing of up
| to 200Gb/s network streams with datacenter gpus (not performing
| inference but I'm talking about actual streaming to gpu memory
| here). PCIe-5 and H100 (the only one in the line-up that has
| PCIe-5...) should also improve on that.
| touisteur wrote:
| Additional note, the L4 inferencing board is also still on
| PCIe-4 (like the L40, very sadly), so NVIDIA doesn't seem to
| foresee the need for more bandwidth on inferencing workloads in
| the next 3 years (until they maybe get a H30 out, with the
| binned not-fully-functional H100s? Maybe? Hopefully?) and/or
| will make you pay for a full H100.
| rain1 wrote:
| why is it called serverless when it is a server
| smw wrote:
| Is it really useful to be pedantic about it? It's a useful term
| at this point because people in the community know what it
| means, no?
| Pulcinella wrote:
| I literally don't know what it means. Does serverless just
| mean doing it locally? Like an on premises cluster of GPUs in
| the IT closet?
|
| If you are doing it on someone else computer, and that
| computer is a big server farm, I find it odd to use the term
| "serverless." And by odd I mean literally the exact opposite
| of serverless.
| CyberDildonics wrote:
| A serverless GPU means a GPU on a server right?
| crazysim wrote:
| Some of these are a bit more host a server for you or others to
| run. I wish this comparison also compared billing models a bit
| and any other value-adds.
|
| The cool thing about replicate.com is that you can use someone
| else's public model and it's billed to the callee. For someone
| like me who is gluing models together for a hobbiest setting,
| it's been great.
|
| For some image identification tasks, it's been pretty neat for me
| to be able to call https://replicate.com/andreasjansson/blip-2
| which is someone else's already deployed model which was already
| pre-warmed by somebody or level of activity, and get results
| back. For me, I've been captioning images and putting them into
| OpenAI prompts.
|
| I've also myself put out
| https://replicate.com/nelsonjchen/minigpt-4_vicuna-13b to see if
| maybe it's an improvement in captioning. Unfortunately, it takes
| like 15 minutes to spin up. That said, it's currently free for me
| to put up. If someone else wants to run it, they can wait/pay.
| And they only pay for the runtime and not setup. And if it were
| to get popular, it'll be naturally warm for everyone. For me, it
| was 6x the cost of https://replicate.com/andreasjansson/blip-2,
| and although my experiment did not produce something suitable or
| usable for me, maybe those caveats are appropriate for someone
| else's use case, and they can super-easily reuse my deployment on
| their dime without costing me any money.
|
| Not to mention that replicate also put out some pretty alright
| APIs or libraries to call their service. It's been consistent.
|
| All this ease of use does come with some caveats, replicate.com
| is pretty expensive for the raw calls.
| 0x445442 wrote:
| Pardon my ignorance on this matter but if the processing unit
| isn't displaying graphics shouldn't it just be referred to as a
| cpu?
| mgraczyk wrote:
| No, "GPU" is a specific piece of hardware.
| https://en.wikipedia.org/wiki/Graphics_processing_unit
| 0x445442 wrote:
| Right, but what graphics are they processing in a serverless
| context or when it's mining bitcoin?
| mgraczyk wrote:
| None, "GPU" is a word with a distinct meaning from "unit
| that processes graphics"
| DonHopkins wrote:
| They're making the line go up when they're mining bitcoin.
|
| https://www.youtube.com/watch?v=YQ_xWvX1n9g
| pdntspa wrote:
| The math is similar so GPUs have been hijacked by AI nerds
|
| Gamers everywhere are pissed that GPUs are so expensive
| now.
| varelse wrote:
| [dead]
| suprfsat wrote:
| If a CPU isn't "central" to much of anything shouldn't it just
| be referred to as a PU?
| 0x445442 wrote:
| Good question.
| dragonwriter wrote:
| > Pardon my ignorance on this matter but if the processing unit
| isn't displaying graphics shouldn't it just be referred to as a
| cpu?
|
| No, i mean, its an auxiliary processor (there is a CPU, and
| this isn't it) doing floating point math, so I guess you could
| call it either a math coprocessor or an FPU, but... we that's
| somewhat confusing for historical reasons.
| etaioinshrdlu wrote:
| It's kind of true, if GPUs were released today they would
| probably be called something like Parallel Processing Units.
| todd3834 wrote:
| This is a great overview! I've been working on a project where
| I've tried a few of these and it's definitely a space where this
| data is super valuable.
|
| I really wanted to like Banana.dev but runpod consistently
| outperforms them for my use case. I love the innovation in this
| space.
|
| Here is my wishlist:
|
| 1. Faster cold starts. If you're building a consumer product that
| has a request and response using one of these services I'm seeing
| at least a 20 second delay before the server begins working.
|
| 2. Much cheaper GPUs. This is an unrealistic expectation right
| now because supply and demand have these services completely
| crowded with people happy to pay. I just wish I could afford to
| have a few of the faster GPUs prewarmed ready to go but that
| would be several thousand dollars a month. Doable eventually but
| for my bootstrapped side project that hasn't found product market
| fit it's a little rough.
|
| 3. Ability to create custom models from the service. I'm on an
| Intel Mac and so making a custom model requires me to ssh into a
| machine. If only there was a service that let me rent a high end
| GPU service by the second. /sarcasm. I wasn't able to get access
| to docker or install it on runpod and support confirmed it isn't
| something they support.
|
| For custom model building I found the prices and flexibility to
| do whatever you need on lambdalabs.com to be best. Also their
| prices are blowing all of these other services away. However no
| serverless option. The space is so crowded with consumers I'm
| almost afraid to even mention them because I worry I won't have
| GPUs available for me. I'm seeing this mentality a lot.
| uvesten wrote:
| The "Executive summary" isn't, it's an abstract.
|
| There is zero actionable information there, it's just describing
| the article. I was hoping to get a recommendation for which
| provider to use for which kinds of workloads. I guess I have to
| read it all then :/
| ec109685 wrote:
| I don't understand how these small companies can compete with the
| big clouds ofer time. They aren't offering anything fundamentally
| different than each other or the big clouds themselves.
|
| At some point, Lambda will offer GPU's on their instances and
| there are plenty of serverless offering right now that have
| GPU's.
|
| Fly.io at least is differentiated with their "run your compute
| closer to the user" with zero hassle. Though even that doesn't
| seem super sustainable in the long term.
| [deleted]
| dragonwriter wrote:
| > At some point, Lambda will offer GPU's on their instances
|
| Lambda does, that's the whole point of the company.
|
| Do you mean AWS will on AWS Lambda?
| ec109685 wrote:
| Yeah, sorry for confusion. AWS Lambda.
| arpowers wrote:
| The large clouds are intentionally hard to use platforms with a
| different target customer than the smaller players ...
| jonfromsf wrote:
| Wow I always thought AWS was crazy complicated just like ..
| because stuff is hard. But you're right, if it's complex
| enough people define their career identity as an "AWS expert"
| and at that point you have lock-in for life.
| cateye wrote:
| As an avid Runpod user, I have come across some information that
| is not entirely accurate.
|
| _- Although the number of models is limited, the platform has a
| community feature where users can fork models._
|
| I'm not entirely sure of the intended meaning, but contradicts
| with being able to bring any container.
|
| However, if it pertains to the selection of pre-made template
| containers, it is true that the options may be limited.
| Nonetheless, there aren't a significant number of commonly-used
| open-source models available either...
|
| Or it is referring to the API endpoints. That is indeed correct
| but it is confusing what this review is exactly about.
|
| _- Post-deployment, it can be confusing to understand how the
| platform works, which may result in users receiving a bill if
| they are not careful._
|
| Although documentation is sometimes lacking, it is very clear
| what it costs and there is not such thing as a bill...
|
| _- There is no bot or instant support mechanism available._
|
| They have a live chat available, and their response time was
| good. They are also very active on their Discord channel,
| providing speedy support to users.
| pavlov wrote:
| The term "serverless GPU" somehow wrecks my brain. Logically the
| absence of a server suggests its opposite, and the opposite of a
| server is a client, and client GPUs are the default. But this
| means "server GPU that's available on-demand for very short-lived
| jobs" I guess.
| ukuina wrote:
| I, too, think of "classical serverless" rather than "neo-
| serverless": https://www.sqlite.org/serverless.html
|
| ------
|
| Recently, folks have begun to use the word "serverless" to mean
| something subtly different from its intended meaning in this
| document. Here are two possible definitions of "serverless":
|
| Classic Serverless: The database engine runs within the same
| process, thread, and address space as the application. There is
| no message passing or network activity.
|
| Neo-Serverless: The database engine runs in a separate
| namespace from the application, probably on a separate machine,
| but the database is provided as a turn-key service by the
| hosting provider, requires no management or administration by
| the application owners, and is so easy to use that the
| developers can think of the database as being serverless even
| if it really does use a server under the covers.
| thomastjeffery wrote:
| Something like "GPU for Serverless computing" would be much
| better.
|
| GPUs don't _contain_ servers. Serverlesses contain GPUs.
| touisteur wrote:
| Yeah, should be 'elastic' and 'very reduced runtime, possibly
| just inferencing', so exposing Triton to an API gateway and
| putting a custom load balancer and task queue facade ? Curious
| too.
|
| Also, GPUs do other things than inferencing, right?
| dvh wrote:
| Serverless = pay per request
|
| Cloud computing = someone's else's computer
| mgraczyk wrote:
| "Serverless" has been a standard term in industry for at least
| 7 years. Sometimes words don't map perfectly onto the
| subcomponents that form them. For example a "mailbox" isn't
| always a box, and doesn't always contain mail. You just have to
| learn to use the words and not worry so much about the
| etymology.
| pavlov wrote:
| The tortured etymology becomes apparent again when these
| words are combined in new ways. "Serverless GPU" might be
| something like "mailbox SSD" in your example. What would that
| mean? It's not obvious at first sight. The metaphor loses its
| power when it's attached to a physical descriptor which is
| not a metaphor.
| mgraczyk wrote:
| No, it's obvious to everyone working with GPUs who knows
| what "serverless" means.
|
| It's a very standard construction in English. "Serverless
| GPU" means "GPU" that is "serverless". If you know what
| both words mean in the jargon, you know what they mean
| together. It's ok to not know what they mean, but arguing
| that it's "tortured" rings to me as misguided obtuseness.
| Mike_12345 wrote:
| > It's a very standard construction in English.
|
| It's a new buzzword coined by the marketing team at
| Amazon in 2014. Somewhat confusing here as you are
| renting time on a GPU server described as "serverless".
| mgraczyk wrote:
| When I said "standard construction", I was referring to
| the combination of "serverless" and "GPU" to make
| "serverless GPU".
|
| Also you're factually wrong. Amazon didn't invent this
| term and it was widely used before 2014. Here's a 2012
| article using the term.
|
| https://readwrite.com/why-the-future-of-software-and-
| apps-is...
|
| I truly don't understand the bizarre and conspiracy-
| theory laden opposition to terms like this. It's a word,
| everyone who matters knows that it means.
| [deleted]
| Mike_12345 wrote:
| > I truly don't understand the bizarre and conspiracy-
| theory laden opposition to terms like this.
|
| I truly don't care and not opposed to the term. Just
| trying to explain why someone might get confused.
| ralusek wrote:
| I don't understand how people haven't gotten over this yet.
| When someone says serverless, I immediately understand that to
| mean "we've obfuscated the underlying server hardware from the
| consumer of this product." It means "you don't think about
| servers," not "there are no servers."
| pavlov wrote:
| I've got used to the term, but when it's in a new context
| like "serverless GPU", its inherent absurdity resurfaces.
| eurekin wrote:
| Yup. My first intuition was "probably some server gpu
| (a100), but sold separately"
| gpm wrote:
| Because if I'm thinking about the _gpu_ I 'm fundamentally
| thinking about the hardware, the server.
|
| Serverless responding to http requests. Sure. I write some
| code. It gets fed data and returns data. I don't have to know
| how many cores the server has, or what microcode version the
| CPU is, or how many other things are running on the server,
| or if I'm writing an interpreted language (probably the case)
| even what architecture the CPU is.
|
| But... I need to know all of that if I'm writing GPU code
| today.
| rcme wrote:
| Are you really thinking about the hardware when thinking
| about the GPU? For instance, if you use pytorch to write a
| NN, don't you kind of expect it to execute on the GPU
| without needing to get into the gritty details of it?
| seabass-labrax wrote:
| The specific version of Nvidia CUDA or AMD ROCm that the
| GPU supports is often very important; some software needs
| to be compiled from a specific branch or with specific
| settings to support a given version. Case in point: the
| official PyTorch website offers four distinct builds for
| various platforms, two of those being different versions
| of CUDA.
|
| I think it'll still be useful for plenty of people to
| choose which GPU runs their code, even if it's compatible
| with any GPU offered by the service. You might want to
| choose an older, cheaper GPU for basic parallel
| computation, but as supply catches up with demand for a
| newer and more energy-efficient model you'll then want to
| switch to that. There are only so many GeForce 4090s to
| go round :)
| faeriechangling wrote:
| I dunno, maybe it's because the jargon is stupid?
| 1letterunixname wrote:
| It comes from the magical thinking of having too many
| marketing people and not enough engineers who actually build
| shit.
| DonHopkins wrote:
| Says the guy whose user name claims to have one letter, but
| actually contains one digit. ;)
| 1letterunixname wrote:
| It's a stupid misnomer. "Cloud" is still another.
|
| Perhaps "distributed scheduling and execution" is too
| straightforward than marketing wank.
| TeMPOraL wrote:
| I mean it's all because marketers didn't want people to think
| of DCOM or CORBA when dealing with this "new" technology.
|
| (I can kind of understand it in case of CORBA - if you put
| the word "broker" in the name of your product, no one else
| will want to associate with it.)
| DonHopkins wrote:
| The term should be "Serverless GPUless" to be fair and
| consistent. What's good for the goose is good for the gander.
___________________________________________________________________
(page generated 2023-04-29 23:01 UTC)