[HN Gopher] Workers AI: Serverless GPU-powered inference
___________________________________________________________________
Workers AI: Serverless GPU-powered inference
Author : jgrahamc
Score : 233 points
Date : 2023-09-27 13:06 UTC (9 hours ago)
(HTM) web link (blog.cloudflare.com)
(TXT) w3m dump (blog.cloudflare.com)
| AmericanOP wrote:
| How does Workers AI compare to Replicate.com?
| edunteman wrote:
| Given this is a hosted API rather than arbitrary hosting, why
| choose the word "serverless"? Do you plan to offer arbitrary
| hosting in the future?
|
| (bias: am Banana CEO)
| bredren wrote:
| I tried serverless for whisper on an existing competitive
| service.
|
| It had a cold boot and run on a 8 word STT time of 45 seconds and
| warm never got past 15 seconds.
|
| This does not work for STT, where it has to be much faster
| turnaround.
|
| Can anyone give feedback on if whisper and any of its model sizes
| can work well on serverless?
|
| Do most AI serverless solutions suffer from significant cold boot
| delays?
|
| The cheapest persistent GPU cloud instance I saw on G was ~$160 a
| month. Is that roughly the kind of money people need to be
| prepared to spend to have a model ready to go at all times as a
| service to another product?
| NicoJuicy wrote:
| Cloudflare has almost no cold boot and I think their ml models
| are prefetched within the same DC, so loading the model the
| first time should have no noticable overhead either.
|
| Correct me if I'm wrong, but that's how I interpreted it when
| they first started with ML models on their CPU's.
| htrp wrote:
| which service did you look into ?
| Mernit wrote:
| Serverless only works if the cold boot is fast. For context, my
| company runs a serverless cloud GPU product called
| https://beam.cloud, which we've optimized for fast cold start.
| We see Whisper in production cold start in under 10s (across
| model sizes). A lot of our users are running semi-real time
| STT, and this seems to be working well for them.
| bredren wrote:
| >...this seems to be working well for them.
|
| Is this because the users are streaming audio in a more
| conversational style?
|
| For example, when you give siri a command, it is stated, and
| then you stop speaking.
|
| For most of ChatGPT's life, in openAI's iOS app, if you
| wanted to speak to input text, you would tap the record
| button, and then tap it off, either using the app's own
| Speech to text capability or siri's input field speech to
| text.
|
| Conversational speech to text is more ongoing, though, which
| would make a 10 second cold start OK, because you don't sense
| as much lag because you're continuing to speak.
|
| Or perhaps people in general record input longer than 10
| seconds, And you are sending the first chunk as soon as
| possible to get whisper going.
|
| Then follow up chunks are handled as warm boots? Then the
| text is reassembled? Is that roughly correct?
|
| Anything you can provide on sort of the request and data flow
| that works with a longer cold boot time in the context of
| single recording versus streaming, and how audio is broken up
| would be helpful.
| FL33TW00D wrote:
| Whisper large is only 1.5B params, why not run it client side
| with something like https://github.com/FL33TW00D/whisper-turbo
|
| (Disclaimer: I am the author)
| bredren wrote:
| Seems like webgpu is not supported by mobile safari yet.
|
| And possibly has coverage in only 65% of the desktop browser
| market. [1] Does that roughly conform to how you understand
| the current penetration of this browser api?
|
| Presuming coverage for a given user, I don't have a good
| answer for why to consider remote.
|
| It seems like it would be worth testing compatibility for
| webgpu and attempting to run on the client if possible, but
| then have a remote instance available otherwise.
|
| Does that make sense to you?
|
| Can you tell me another reason why someone would want a
| remote instance of whisper given 20x realtime potential at
| client in your project?
|
| [1] https://caniuse.com/webgpu
| FL33TW00D wrote:
| Indeed WebGPU is basically only supported on Chromium based
| browsers.
|
| This means that the primary usecase for whisper-turbo and
| my upcoming libraries is Electron/Tauri apps. For users
| that don't have WebGPU supported for whatever reason, we
| will still hit OAI/other server deployment. In the ideal
| case there should be a 90% cost reduction and same or
| improved UX.
|
| Someone will still want a remote instance today as there is
| still engineering to be done. I need more aggressive
| quantization, better developer experience and more features
| in order to get people off of the OAI API.
| chatmasta wrote:
| Maybe the client is another backend service or serverless
| function, i.e. one where they'd need to pay for the GPU
| anyway.
| fragmede wrote:
| (STT: Speech to Text)
| jonplackett wrote:
| Very cool and also very simple as I'd expect from Cloudflare.
|
| But I have a question - why not make inference as easy as the
| translation? Why do I have to run that in a worker rather than
| just as a simple API call? That would be much simpler.
|
| Is there a technical reason or is it that people would want to
| have logic before making the call to llama?
| rita3ko wrote:
| you can do both! all of our models are supported either via
| workers / pages binding (which makes it really easy to host the
| rest of the logic), and via REST API
|
| docs: https://developers.cloudflare.com/workers-ai/get-
| started/res...
|
| (llama specific example here too under curl:
| https://developers.cloudflare.com/workers-ai/models/llm/ )
| jonplackett wrote:
| Awesome! Thanks for that. Cloudflare smashing it on
| simplicity as per usual.
| winddude wrote:
| kinda cool, but rather limited if you can't use custom models.
| jgrahamc wrote:
| You will be able to.
| winddude wrote:
| looking forward too it!
| mistrial9 wrote:
| [flagged]
| willquack wrote:
| I've never played with Cloudflare workers, but I thought they
| were implemented as JavaScript runtimes that form an edge
| computing network
|
| Are the models run in JavaScript/WebAssembly behind the scenes?
| celso wrote:
| You can use Javascript or Wasm to interface with the AI
| binding, think of it of as the SDK, but the inference task
| itself runs natively on top of a ML runtime and the models are
| loaded into GPUs.
| jgrahamc wrote:
| TL;DR: GPUs all over the Cloudflare global network; working
| closely with Microsoft, Meta, Hugging Face, Databricks, NVIDIA;
| new Cloudflare-native vector database; inference embedded in
| Cloudflare Workers; native support for WebGPU. Live demo:
| https://ai.cloudflare.com/
| aprxi wrote:
| Is CodeLlama somewhere on the roadmap?
| tebbers wrote:
| Hey John, great work on this! Just a headsup, small typo on
| that page under R2: "Build mutli-cloud training architectures
| with free egress."
| foggedb0nk wrote:
| Any chance you're looking for technical product folks to work
| on this? I actually worked on a very similar deployment
| internally at Livepeer (focus was on live video enhancements
| but also generalized edge compute)!
| rita3ko wrote:
| we always are! email is rita at cloudflare dot com :)
| ushakov wrote:
| Do you actually run the inference in the worker? Or is it like
| what Fermyon does where they basically host the models for you
| and you get a SDK that is automatically connected to the
| function?
| celso wrote:
| Unlike the first version of Constellation, Workers AI runs
| inference directly on GPUs that we are (quickly) installing
| in our global network.
| ushakov wrote:
| But the code isn't running on the worker? It runs somewhere
| else on a GPU cluster?
| jgrahamc wrote:
| It's a little like how Cloudflare Workers runs. You don't
| know which CPU it runs on, all you know is it's a CPU
| close to your end user. Same goes for this. We are
| rolling out GPUs everywhere across the globe and so
| Workers AI will just use a nearby GPU. Probably in the
| same machine as your workers, or maybe the same data
| center, or whatever other smart routing decision we make.
| What we are not doing is running a massive GPU cluster
| somewhere. This is all distributed and that's the power
| of owning your own network.
| pseg134 wrote:
| Since they don't seem to be able to give a simple answer:
| the inference does not run in the worker. It connects to
| external GPUs.
| eastdakota wrote:
| I think the confusion is what is meant by "in the
| Worker." From a hardware perspective, the GPU may be in
| the same machine as the CPU that's powering the Worker.
| Or they may be across different machines in our network.
| We are not routing requests to some third party. And we
| will try to run the inference task as close as possible
| to who/whatever requested it. The whole idea of
| "serverless" is you shouldn't have to worry about what
| machine where runs whatever unless you're on the team
| building the scheduling and routing logic at Cloudflare.
| thegagne wrote:
| I think his question is more about does the worker
| directly access the GPU and thus require js tooling to
| handle the GPU somehow (no), or does it make subrequests
| to a separate GPU service not running the worker runtime
| (yes).
| claytonjy wrote:
| I see plans for more models via HF partnership, but can I or
| will I be able to run a custom fine-tuned version of a
| supported model?
| celso wrote:
| On top of our hosted and supported catalog of models, and the
| deploy to CF partnerships like the HF one, you will also be
| able to bring your own custom model at some point in time.
| claytonjy wrote:
| Awesome. What about compiled model support? Running most of
| the listed models without compilation only makes sense for
| hobby projects.
| ryandetzel wrote:
| I want to love workers but I've never had great luck with anymore
| more than a basic crud app. Even getting an external db to
| connect proved to be more work than it should have, and their
| docs are all outdated and all over the place, often contradicting
| themselves.
| throwaway77384 wrote:
| The docs are soooooo bad. This is the same story with any
| exciting / new product.
|
| Honestly, I'm beginning to think that we need some kind of
| documentation-first style development. Like TDD, but DDD....
|
| I have integrated Facebook APIs, Instagram (well, same thing),
| Google APIs, Stripe APIs, Mailchimp APIs, etc. etc.
|
| And the only thing common among all of them? The documentation
| is _terrible_..., like, _terrible_.
|
| I also run a few products online and I spend so, so much time
| trying to get the documentation right. It's incredibly boring
| and tedious, but I really feel that if you want to set yourself
| apart from the big players, make good documentation. It can't
| be that hard.
| kristopolous wrote:
| good documentation is the key to php's success along with a
| bunch of other things that get dismissed as "inferior
| technology".
|
| The worst documentation are the jargon filled abstract vibes-
| based ones where the authors basically typed it with one hand
| on the keyboard. It's like "ok, you're amazing. Now how do I
| resolve this error and what are your command line flags?"
| [deleted]
| amayne wrote:
| This is very cool. I'm still trying to understand the pricing.
| What is a "neuron" in this context? A token? A character?
|
| "Neurons are a way to measure AI output that always scales down
| to zero (if you get no usage, you will be charged for 0 neurons).
| To give you a sense of what you can accomplish with a thousand
| neurons, you can: generate 130 LLM responses, 830 image
| classifications, or 1,250 embeddings."
|
| 130 LLM responses of that length? 1,250 embeddings what size of
| text?
| spikey_sanju wrote:
| You can see the limitations here--
| https://x.com/spikeysanju/status/1707057365231812630?s=46
| amayne wrote:
| Those don't explain the relation between neuron cost and
| length.
| eastdakota wrote:
| It's effectively a unit of time benchmarked to what we can
| accomplish in that time as of Sept 27, 2023 (launch). The
| challenge here is that because we're abstracting away the
| underlying hardware it's not the same as renting a VM for a
| period of time. We also don't want to create perverse
| incentives that keep us from making the underlying system
| faster. It's similar to how AWS standardized EC2 to a standard
| compute unit. Over time, as we continue to add faster and
| faster hardware and better optimize models we expect the cost
| of a neuron will trend down but the amount of AI inference work
| that you can do with a neuron will remain relatively constant.
| choppaface wrote:
| Really amazing stuff to see this launch with Hugginface! Hope
| to see it expand beyond text too.
|
| "neuron" is a cute name but there's too much conceptual
| overlap with floating point ops, layers, model parameters etc
| which are time independent. Should just call them inference
| credits or something. When some large model runs on multiple
| GPUs it's even more confusing what neurons / dollars per
| second might be.
| amayne wrote:
| Could you give us an example of what that means in practical
| terms?
|
| The post says that 1000 neurons will give you 130 LLM
| responses - but of what length?
|
| (LLMs are generally priced by input and output tokens. The
| longer the tokens the longer the compute time. Without an
| idea of what you mean by a response it's hard to understand.)
|
| Likewise: 1,250 embeddings - How big is the text size in the
| example?
|
| I'm VERY excited to see you doing this and understand it's
| early stages, but I wan't wrap my head around the pricing
| without context.
| KRAKRISMOTT wrote:
| Please rename it, or at least make sure it corresponds to
| actual neural operations. It's terribly confusing for
| practitioners.
| coolspot wrote:
| Sounds like 1 Neuron ~= X FLOPS
| esafak wrote:
| Then call it something like Neural Time Unit (NTU) or
| Computational Time Unit (CTU) because neurons make people
| think of neural networks. As in, you pay for the size of your
| model.
| thyrox wrote:
| This would be a game changer if they had something for image
| generation as well. Oh well, maybe it's coming soon as the page
| says it's just a small preview. The best part for me personally
| is it's available on all plans and pricing looks good too.
|
| OT but if cloudflare fixes their false positives on showing
| random captchas when I'm trying to browse the net on VPN they
| will surely be one of my favorite companies.
| celso wrote:
| Image generation is coming.
| xfalcox wrote:
| Embedding cost and model choice makes this a very compelling
| choice. I'm working on leveraging embeddings in
| https://github.com/discourse/discourse-ai where it powers
| offering related topics, semantic search, tag and category
| recommendations among other things.
|
| A cheap offering like this can make it a lot more reasonable for
| self-hosters.
| baobabKoodaa wrote:
| This could become something useful in the future, but right now
| it appears to be toy models only. I'm assuming Cloudflare will
| add useful models eventually, but the cold start times are going
| to be horrible on those. I'm struggling to think of useful
| applications for this. Maybe one day.
| celso wrote:
| Which models would you find useful?
| NicoJuicy wrote:
| At this cost, no one will be running embeddings elsewhere :o
|
| https://twitter.com/eastdakota/status/1707056412575023352?t=...
| arbitrarian wrote:
| It will be interesting to see if they can undercut OpenAI
| themselves on the cost for running Whisper in the cloud.
| bobjmiles wrote:
| https://blog.salad.com/whisper-large-v2-benchmark/
| showerst wrote:
| I tried to spin up a free plan and run the whisper demo on a new
| worker and it immediately just gives me an:
| Error 1102 Worker exceeded resource limits
|
| Did I mess up the config or is it just not intended to be tried
| out without being on a paid plan already?
| pdwittig wrote:
| It's 100% designed to let you try it out for free, so something
| else must be going on. Feel free to message me at pwittig at
| cloudflare dot com, and I'm happy to help debug.
|
| Also, we're still figuring some things out, but current limits
| are here: https://developers.cloudflare.com/workers-
| ai/platform/limits...
| [deleted]
| rtcode_io wrote:
| We were building AI https://efn.kr/#ai into https://RTCode.io and
| ...
|
| Cloudflare drops this! Sweet! Now, we have BYOAI.
|
| Our whole offering runs on their network https://RTEdge.net
|
| Our playground lets one live-code user workers that deploy to
| Cloudflare Worker for Platforms!
|
| - https://sw.rt.ht/?io (in-browser)
|
| - https://sw.rt.ht/ (region-Earth)
| ignoramous wrote:
| All well and good, but why is the webpage loading this
| obfuscated javascript file? https://archive.is/htQgN
| cetinsert wrote:
| CEO here, simply because we want to protect the core
| components that differentiate our services. Similar to, say
| https://www.photopea.com/ and many others we can find if we
| look behind the scenes.
|
| Once we raise funding, and establish strong market presence,
| we will revisit this decision, and dedicate developer
| resources to sharing our in-house tech with the world more
| openly. This will take a full position. If you liked what you
| see, and want to work with us, send us an email at
| work@elefunc.com and we will get in touch once we have open
| positions!
| [deleted]
| seejayseesjays wrote:
| It would be super cool to see SD running on this! hyped to play
| around with llama since I don't have access to a good GPU.
| latchkey wrote:
| Interesting how the same exact blog url was used previously in
| April 2021.
|
| https://news.ycombinator.com/item?id=26795517
| eastdakota wrote:
| That was our early alpha cooperation with NVIDIA. We've learned
| a lot since then. Not to mention, the AI ecosystem has grown up
| a bunch. But you are correct: this is something we've been
| planning for for a loooooooong time.
| latchkey wrote:
| I'd love to see a blog post about the knowledge delta and
| growth progress. That's more interesting to me than the
| actual announcements.
| siwakotisaurav wrote:
| The biggest one missing is stable diffusion.
| rita3ko wrote:
| stay tuned!
| [deleted]
| foggedb0nk wrote:
| Any chance you're looking for technical product folks to work
| on this? I actually lead a very similar deployment internally
| at Livepeer (focus was on live video enhancements but also
| generalized edge compute)!
| jgrahamc wrote:
| You can always email jgc@cloudflare.com and I'll route the
| resume to the right people.
| UnlockedSecrets wrote:
| I am having quite a few issues getting the reference API code at
| https://developers.cloudflare.com/workers-ai/models/llm/ to work?
|
| {'errors': [{'code': 'invalid_union', 'unionErrors': [{'issues':
| [{'code': 'invalid_type', 'expected': 'object', 'received':
| 'string', 'path': ['body'], 'message': 'Expected object, received
| string'}], 'name': 'ZodError'}, {'issues': [{'code':
| 'invalid_type', 'expected': 'object', 'received': 'string',
| 'path': ['body'], 'message': 'Expected object, received
| string'}], 'name': 'ZodError'}], 'path': ['body'], 'message':
| 'Invalid input'}], 'success': False, 'result': {}}
| 0xDEF wrote:
| How well-proteced is the "edge" in edge computing? I can see
| Cloudflare has edge locations in many countries. Can entities
| with physical access to Cloudflare's edge machines get access to
| sensitive user data?
| r1ch wrote:
| With Cloudflare's default settings, a malicious entity can
| intercept any Cloudflare <-> Backend connections invisibly to
| the end user since the SSL certificates aren't validated. The
| end user also can be victim to plain old HTTP MITM on
| Cloudflare's upstream networks, as happened in 2016:
| https://news.ycombinator.com/item?id=12091900
|
| It's hard to take Cloudflare's commitment to security seriously
| when they still ship such terrible default settings.
| NicoJuicy wrote:
| What do you mean?
|
| You can install certificates by cloudflare and then the only
| one that can connect to your server is from cloudflare.
|
| No one can intercept it then.
|
| If you're talking about flexible SSL. Sure, you can use it
| purely as a https proxy for your SEO score of your blog. But
| securing it is not much effort.
|
| If it's just for a static blog, I'm not sure what you would
| though.
| r1ch wrote:
| If you have a valid HTTPS certificate for example.com and
| then add example.com to Cloudflare, your overall security
| decreases because the path from the CF datacenter to your
| origin is now vulnerable to MITM - the default SSL setting
| is "Full" which doesn't check certificate validity.
|
| To the less experienced sysadmin everything looks like it's
| working fine and users also don't notice any difference,
| which is why it's a terrible default.
|
| Sure you _can_ configure Cloudflare securely, but it should
| be secure out of the box. But that adds friction when the
| origin doesn't have a valid SSL certificate which probably
| hurts someone's KPIs.
| celso wrote:
| https://blog.cloudflare.com/securing-memory-at-epyc-scale/
| https://blog.cloudflare.com/speeding-up-linux-disk-encryptio...
___________________________________________________________________
(page generated 2023-09-27 23:01 UTC)