[HN Gopher] Breaking up with Flask and FastAPI: Why they don't s...
___________________________________________________________________
Breaking up with Flask and FastAPI: Why they don't scale for ML
model serving
Author : yubozhao
Score : 30 points
Date : 2022-06-16 18:26 UTC (4 hours ago)
(HTM) web link (modelserving.com)
(TXT) w3m dump (modelserving.com)
| [deleted]
| anderskaseorg wrote:
| > _While FastAPI does support async calls at the web request
| level, there is no way to call model predictions in an async
| manner._
|
| This confuses me. How is that FastAPI's fault? Can't you just
| asynchronously delegate them to a
| concurrent.futures.ThreadPoolExecutor or
| concurrent.futures.ProcessPoolExecutor? What does Starlette
| provide here that FastAPI doesn't? If the FastAPI limitations are
| due to ASGI, shouldn't Starlette have the same limitations?
| timliu99 wrote:
| > While there are several different methods to use your own
| executor pools or potentially use shared memory for a large
| model, all of these solutions are not first-class solutions for
| ML use cases...
|
| Definitely not FastAPI's fault and yes Starlette has the same
| limitations. BentoML builds additional ML features/abstractions
| on top of Starlette. We introduced a "runner" concept which
| automatically creates separate processes for models to run in.
| anderskaseorg wrote:
| Great--but then one ought to be able to performantly use that
| same runner equally well from FastAPI, Starlette, Quart (ASGI
| port of Flask), or any other ASGI framework. You've decided
| to build a convenient integration with Starlette instead of
| the others, but it's weird to frame this as an argument that
| other frameworks are ill suited for this domain.
| timliu99 wrote:
| Wait... so you're tell me FastAPI is slow...
| jeremycarter wrote:
| Actually it's very slow. Wait until you have 50 concurrent
| customers hitting it.
| nemexis wrote:
| That's weird, it is supposed to be faster than alternatives.
| Do you have some benchmarks to support that critique?
| ssheng wrote:
| FastAPI is great building block but can't expect it to work for
| model serving out of box.
| beckingz wrote:
| "FastAPI is not perfect for ML serving"
|
| Yup. There's a huge amount of work that you need to do to do the
| whole ML lifecycle, and FastAPI doesn't support that out of the
| box like a full fledged ML Platform.
|
| But you probably don't actually want a full ML Platform because
| they're all opinionated and if you try and fight them it's often
| worse than just serving it as an API via FastAPI...
| alpineidyll3 wrote:
| Yeah... I mean the author is blatantly biased, but his low
| level discussion is cogent.
| detroitcoder wrote:
| The jump of going from model -> webserver by placing the
| webserver in the same process as the model is enticing because
| you can get it to work in under an hour by adding
| flask/django/fastapi to the env and decorating a function. The
| problem is that that your model and webserver do NOT scale in the
| same way, and if you don't realize this fast, you are going to be
| trying to fit a square peg through a round hole once you have
| adoption trying to make it work.
|
| All models at scale eventually need to be executed by an async
| queue processor which is fundamentally different from a request
| response REST API. For simplicity managing this outside of the
| process making the web request will help you debug issues when
| people start asking why they are getting 502 responses. If you
| are forced to use python for this, I would always suggest of
| going to celery/huey/dramatiq as an immediate next step after the
| REST API MVP. I hear Celery is getting better but I have ran into
| issues over the year so it pains me to recommend it.
| kroolik wrote:
| Exactly this.
|
| Prime example of the difference is whether you accept
| disruption of inference when you deploy a new version of your
| webapp.
|
| Very high chances you don't, thus you will start implementing a
| queueing mechanism without realizing it.
| ttymck wrote:
| Forgive me, I don't mean this flippantly, but it sounds like you
| implemented queuing and multiprocessing consumers on a Starlette
| webserver. "micro batching" is a feature enabled by the queueing.
| The GPU/CPU abstraction is nice, but I feel it's buried by the
| "FastAPI isn't good enough" digression. If it were framed as
| "here's what we added to the Starlette ecosystem", I would have
| approached it much more agreeably.
|
| It would've been delightful to see "instantiate a runner in your
| existing Starlette application". I don't want to instantiate a
| Bento service. Perhaps I can mount the bento service on the
| Starlette application?
|
| Apologies if I am still grossly misunderstanding. I tried to look
| through some of the _internal codebase to see how the Runner is
| implemented, the constructor signatures are very complex and the
| indirection to RunnerMethod had me cross-eyed.
| sauyon wrote:
| You can absolutely mount a BentoML service into your own
| Starlette (or any ASGI framework); `svc.asgi_app` is all you'd
| need.
|
| Instantiating and using a runner can be done anywhere with
| `init_local`, but it's really the runner ASGI app that does the
| work of queuing and batching. We've thought about allowing
| users to spin that app up separately but it's not a focus right
| now; instead we're trying to ensure that the system is as easy
| to use for data scientists as possible and have that workflow
| fully ironed out before we support the more advanced use-cases.
|
| The whole runner situation is quite complex because we wanted
| to support user-created runners in the nicest way possible, and
| also leave the space open for non-python runners (and service
| app) in the future.
| lmeyerov wrote:
| We have been using async python for GPU pydata , including
| fronting dask/dask_cuda for sharing and bigger-than-memory
| scenarios, so a lot rings true.
|
| For model serving, we were thinking Triton (native vs python
| server) as it is a tightly scoped problem and optimized: any perf
| comparison there?
| andrewstuart wrote:
| FastAPI is collapsing under the weight of its github issues
| (1,100) and pull requests (483).
| ttymck wrote:
| Could you clarify what you mean by "collapsing under the
| weight"?
|
| Will a project be abandoned because users are pointing out ways
| to improve it? Does work stop at some point because "we can't
| get to every issue"? Sure, the maintainer could get burned out,
| but that is not a given.
| agumonkey wrote:
| IIRC FastAPI is built on starlette, can some of these issues
| belong to them ? just hoping the numbers don't reflect a whole
| reality.
| pratikss wrote:
| Simply could also mean it's thriving. FastAPI has nearly 50k
| stars. It is very popular and very effective in what it
| promises. Active repos with similar popularity have way, WAY
| more issues and pull requests, projects like Apache/echarts,
| Godot, Redis or Grafana.
|
| Projects with similar stars, momentJS or Jquery don't see much
| active development have very few issues and pull requests.
| isoprophlex wrote:
| Nice advertorial but what about a a queue and some machines
| running torchserve?
| kroolik wrote:
| I have a feeling the root cause is not in FastAPI or Flask, but
| in the architecture of the system itself.
|
| Why? You are doing the inference in the same request, which is
| synchronous from the perspective of the caller. The request can
| be memory-intensive or cpu-intensive. And the issue is you can't
| efficiently consider all the workloads for a single machine
| without being bottlenecked by Python.
|
| I would say that the problem is in your approach trying to use
| the webapp hammer for all the different flavors of nails in your
| system, using a language that isn't suited for concurrency. What
| I would do is decoupling the validation/interface logic from your
| models via a queue. This way you can scale your capacity
| according to workload and make sure the workload runs on hardware
| most relevant to the job.
|
| I have a feeling trying to throw a webapp at the problem might
| not solve your root issue, only delay it in time.
| saltedonion wrote:
| Apologies for the noob question but how would FastAPI/Flask
| know that the job has been successfully completed? Would the
| worker have to persist ml inference results somewhere and the
| FastAPI server poll it periodically?
| kroolik wrote:
| This is a very good question, with a lot of different answers
| depending on your use-case.
|
| One approach is to translate the synchronous call into an
| async call plus polling on the webapp side. You push onto a
| queue, with the callback queue in the message body. But that
| gives you problems when you want deploy a new version of your
| webapp - existing connection will be disrupted and the state
| lost.
|
| Since you need to deal with retries, anyway, you can move the
| logic into the client itself. It will get the request id on
| the initial response and then ask the service for results.
|
| You see, this solution can vary wildly depending on your
| scalability, durability and resiliency requirements. And on
| your budget. Its not wild to expect the response to be big,
| so you might want to upload it to s3. You might use
| websockets, too. Technology gives you a lot of options here,
| of different levels of complexity and scalability ;)
| heavyset_go wrote:
| Might be a good application of WebSockets or server-sent
| events with a job queue on the backend. That or polling on
| the client-side.
| sauyon wrote:
| If I had my way both Python runners and the runner webapp would
| not be long for this world.
|
| I will say, though: the runner webapp MVP exists to do
| basically what you're describing (and keeps an internal queue).
| Yatai is architected so that the runner instances run on a
| separate Kubernetes cluster to the pre- and post-processing
| code that that's run in the main webapp itself, and can be
| scaled separately.
| jononor wrote:
| Having designed similar systems, I agree completely. This is
| not a language or framework problem, but system architecture. I
| have used RabbitMQ to great success as the message queuing
| system, and there are many other good alternatives.
| higeorge13 wrote:
| I wouldn't put the model under fastapi or any similar
| framework, i would serve it from a different process to also
| allow me to serve multiple versions of the model as well
| (similar to tf serving). But eventually we have an API call
| to some web framework calling this different process and
| requiring a response with the model recommendations to be
| returned in a few milliseconds, how is a message queue
| appropriate for such a real-time use case, could you
| elaborate?
| kroolik wrote:
| Would you describe getting response from the model within
| few milliseconds cpu or memory intensive? What I assumed as
| to the characteristics of inference is a process that takes
| multiple seconds to minutes.
|
| In a system that requires a response to the customer within
| a few milliseconds through web api, how do you ensure the
| performance in Python? I'm genuinely interested as that
| sounds outside of what stock CPython can do except trivial
| logic.
|
| Another aspect that's completely ignored are the
| requirement - you might need the response time in
| milliseconds, couple of times per day. Others might need to
| serve hundreds of requests. You also need to consider the
| budget available.
| sgt101 wrote:
| Advert disguised as experience report.
| timliu99 wrote:
| Disclaimer: I'm the author.
|
| We've been using Flask for years as the foundation for our 0.13
| version. Our choice to move away from Flask and FastAPI as a
| core part of our library is based on our experience with
| hundreds of users and use cases
| Sin2x wrote:
| So why not Quart if you used Flask?
___________________________________________________________________
(page generated 2022-06-16 23:02 UTC)