[HN Gopher] Show HN: Faster FastAPI with simdjson and io_uring o...
___________________________________________________________________
Show HN: Faster FastAPI with simdjson and io_uring on Linux 5.19
A few months ago, I benchmarked FastAPI on an i9 MacBook Pro. I
couldn't believe my eyes. A primary REST endpoint to `sum` two
integers took 6 milliseconds to evaluate. It is okay if you are
targeting a server in another city, but it should be less when your
client and server apps are running on the same machine. FastAPI
would have bottleneck-ed the inference of our lightweight UForm
neural networks recently trending on HN under the title "Beating
OpenAI CLIP with 100x less data and compute". (Thank you all for
the kind words!) So I wrote another library. It has been a while
since I have written networking libraries, so I was eager to try
the newer io_uring networking functionality added by Jens Axboe in
kernel 5.19. TLDR: It's excellent! We used pre-registered buffers
and re-allocated file descriptors from a managed pool. Some other
parts, like multi-shot requests, also look intriguing, but we
couldn't see a flawless way to integrate them into UJRPC. Maybe
next time. Like a parent with two kids, we tell everyone we love
Kernel Bypass and SIMD equally. So I decided to combine the two,
potentially implementing one of the fastest implementations of the
most straightforward RPC protocol - JSON-RPC. ~~Healthy and Fun~~
Efficient and Simple, what can be better? By now, you may already
guess at least one of the dependencies - `simdjson` by Daniel
Lemiere, that has become the industry standard. io_uring is
generally very fast, even with a single core. Adding more polling
threads may only increase congestion. We needed to continue using
no more than one thread, but parsing messages may involve more work
than just invoking a JSON parser. JSON-RPC is transport agnostic.
The incoming requests can be sent over HTTP, pre-pended by rows of
headers. Those would have to be POSTs and generally contain
Content-Length and Content-Type. There is a SIMD-accelerated
library for that as well. It is called `picohttpparser`, uses SSE,
and is maintained by H2O. The story doesn't end there. JSON is
limited. Passing binary strings is a nightmare. The most common
approach is to encode them with base-64. So we took the Turbo-
Base64 from the PowTurbo project to decode those binary strings.
The core implementation of UJRPC is under 2000 lines of C++.
Knowing that those lines connect 3 great libraries with the newest
and coolest parts of Linux is enough to put a smile on my face.
Most people are more rational, so here is another reason to be
cheerful. - FastAPI throughput: 3'184 rps. - Python gRPC
throughput: 9'849 rps. - UJRPC throughput: -- Python server with
io_uring: 43'000 rps. -- C server with POSIX: 79'000 rps. -- C
server with io_uring: 231'000 rps. Granted, this is yet to be your
batteries-included server. It can't balance the load, manage
threads, spell S in HTTPS, or call parents when you misbehave in
school. But at least part of it you shouldn't expect from a web
server. After following the standardization process of executors
in C++ for the last N+1 years, we adapted the "bring your runtime"
and "bring your thread-pool" policies. HTTPS support, however, is
our next primary objective. --- Of course, it is a pre-production
project and must have a lot of bugs. Don't hesitate to report them.
We have huge plans for this tiny package and will potentially make
it the default transport of UKV: https://github.com/unum-cloud/ukv
Author : ashvardanian
Score : 241 points
Date : 2023-03-06 15:41 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| billconan wrote:
| I'm curious if json + simdjson will outperform protobuf.
|
| I'm picking a protocol for my project. I was looking at protobuf,
| but this post made me think otherwise.
| neonsunset wrote:
| Most protobuf implementations are unfortunately not a good
| example of high-performance binary serialization code (but
| still better than the average JSON serialization library! And
| Go, C# (it provides protobuf almost out of box), Java and Rust
| implementations appear to be decent).
|
| However, your use case is likely doesn't need to process multi-
| gigabit traffic in JSON requests. Therefore, protobuf, or more
| specifically, gRPC will be just fine.
| sigg3 wrote:
| Speaking of Go, there's a simdjson implementation for golang
| too:
|
| > Performance wise, simdjson-go runs on average at about 40%
| to 60% of the speed of simdjson. Compared to Golang's
| standard package encoding/json, simdjson-go is about 10x
| faster.
|
| I haven't tried it yet but I don't really need that speed.
|
| https://github.com/minio/simdjson-go
| ashvardanian wrote:
| Yes, even though protobuf is a binary format, it is slower to
| parse than JSONs with simdjson. Counterintuitive, but that is
| the power hardware acceleration :)
| billconan wrote:
| Thank you! Yes, I saw a benchmark somewhere else that also
| says simdjson is faster than protobuf.
| dom96 wrote:
| 100x faster than FastAPI seems easy. I wonder how it compares to
| other fast Python libraries like Japronto[1] and non-Python ones
| too.
|
| 1 - https://github.com/squeaky-pl/japronto
| neonsunset wrote:
| It's hard to beat SimdJSON + io_uring. Other implementations
| should provide a port of the former or equivalent/faster
| implementation and too rely on io_uring, with epoll being a
| viable competitor in limited set of scenarios. I would expect
| Python being also a bottleneck here with other compiled
| languages (besides C/C++) that have good interop and ability to
| write vectorized code having an upper hand like Rust, C# or Go.
| ashvardanian wrote:
| You are right! For the convenience of Python users, we have
| to introspect the messages and parse JSON into Python
| objects. Every member of every dictionary being allocated on
| heap.
|
| To make it as fast as possible we don't use PyBind, NanoBind,
| SWIG, or any high-level tooling. Our Python bindings are a
| pure CPython integration. There is just no way to beat that
| combo, not that I know.
|
| https://github.com/unum-cloud/ujrpc/blob/main/src/python.c
| ashvardanian wrote:
| The difference would still be huge. It's exceptionally large
| even when we compare to the gRPC C++ server implementations.
| dom96 wrote:
| How large? Also I'm not sure the gRPC C++ server
| implementations you've tested are the fastest. If you're
| comparing to FastAPI (which is more of an HTTP server
| framework) then you should also compare to what is at the top
| of https://www.techempower.com/benchmarks/#section=data-r21.
| eatonphil wrote:
| On a tangent, does anyone have code samples of parsing large JSON
| newline files with simdjson that don't fit into memory? I.e.
| where the file overall is many GB (more than 4) but individual
| documents in the top-level are not that big. Most of the code
| samples for simdjson assume you can load the whole file in
| memory.
|
| I've got an idea for how you can do it with
| parse_many(json,window) and truncated_bytes() but it would be
| easier if there were just an example out there I could look at.
|
| See pages like [0] and [1] that describe it should be possible
| but I am just not seeing (or yet able to produce myself) working
| code.
|
| [0] https://github.com/simdjson/simdjson/issues/188
|
| [1]
| https://github.com/simdjson/simdjson/blob/master/doc/iterate...
| saidinesh5 wrote:
| At my last job, we have had to deal with this. We had huge json
| files (compressed size of 20+ GB) and had to perform certain
| operations on the data. I found that rapidjson had a better
| (not perfect, just better) API to do this. I quickly wrote a
| little wrapper around it to process json data coming in as
| streams. This was a writeup about it:
| https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ .
|
| If there is any interest, I can ask if they can to open source
| it.
| sl-dolt wrote:
| Absolutely interested, on my end at least. I wrote this to
| manage the transparency in coverage files:
| https://github.com/dolthub/data-
| analysis/tree/main/transpare... but I'm always looking for
| better techniques.
|
| Edit: Oh wow, I see you used it on those exact files. How
| about that.
| tinselcity wrote:
| I used the rapidjson streams with my little embedded REST
| HTTP(s) server library: https://github.com/Edgio/is2/
|
| We needed it for streaming large json from async server
| sockets.
|
| Code link: https://github.com/Edgio/is2/blob/master/include
| /is2/support...
|
| You just had to implement the interfaces like
| Peek/Take/Tell/etc. It worked really well for us.
|
| Probably not as fast as simdjson, but they used some simd
| tricks I think for skipping whitespace:
|
| https://rapidjson.org/md_doc_internals.html#SkipwhitespaceW
| i...
| saidinesh5 wrote:
| Ha! Thanks to you, Today I found out how big those
| uncompressed JSON files really are (the data wasn't
| accessible to me, so i shared the tool with my colleague
| and he was the one who ran the queries on his laptop):
| https://www.dolthub.com/blog/2022-09-02-a-trillion-prices/
| .
|
| And yep, it was more or less they way you did with ijson. I
| found ijson just a day after I finished the prototype.
| Rapidjson would probably be faster. Especially after
| enabling SIMD. But the indexing was a one time thing.
|
| We have open sourced the codebase. Here's the link:
| https://github.com/multiversal-ventures/json-buffet . Since
| this was a quick and dirty prototype, comments were sparse.
| I have updated the Readme, and added a sample json-fetcher.
| Hope this is more useful for you.
|
| Another unwritten TODO was to nudge the data providers
| towards a more streaming friendly compression formats - and
| then just create an index to fetch the data directly from
| their compressed archives. That would have saved everyone a
| LOT of $$$.
| aobdev wrote:
| Just to be clear, are you talking about a file where each line
| is its own json object rather than the entire file being one
| large array?
|
| If so, it's easy in Python to do the following:
| with open("file.json") as src: for line in src:
| json.loads(line)
| eatonphil wrote:
| I'm talking about with simdjson. Lemire suggested reading
| line-by-line is not a good idea [0]. So I'm asking about the
| ideal approach using simdjson, not JSON parsers in general.
|
| [0] https://github.com/simdjson/simdjson/issues/188#issuecomm
| ent...
| MikeDelta wrote:
| We had to parse thousands of multi GB zipped jsons for
| financial data. I don't have any code but it involved (in
| C++) boost gzip to unpack chunks into 100MB blocks and the
| simdjson iterate_many function to parse the block into a
| stream. The json files were not line-delimited but every
| row had a newline (json: [\n{...},\n{...}\n....\n] ), so we
| had to clean the commas in order for simdjson to process it
| as if it were newline-delimited.
|
| Whatever remained at the end of the buffer (few hundred
| bytes of incomplete json) we would copy just before the
| boost-unzip block so that there was a continuation of json
| data. We also reused the parser and string buffer for
| performance.
|
| Also try to parse the json fields in the right order for
| fastest performance, if possible.
| ashvardanian wrote:
| There is a simple way and there is a hard way.
|
| Simple - memory-map the whole file with `mmap`. We also suggest
| using `madvice`, to inform the kernel that you will be
| accessing the data just once, strictly in the sequential order.
| And then simply give it to SIMDJSON. With `ondemand` parser,
| your memory usage will be proportional to the depth of the
| deepest tree, not their number.
|
| Harder way, would be to parse file in chunks, locating
| continuous sequences that form a single document.
|
| There may be more options, but I'd have to check the docs :)
| roseway4 wrote:
| Regarding the hard way, this little utility does a great job
| of splitting larger than memory JSON documents into
| collections of NDJSON files:
|
| https://github.com/dolthub/jsplit
| eatonphil wrote:
| Thanks! Yeah I've seen both general advice but since I'm not
| incredibly familiar with the API I've had a hard time
| creating code that works.
|
| If you have any code examples of either of those options I'd
| love to see them!
|
| Edit: Also, I'm curious: what are the downsides to doing the
| mmap route?
| loeg wrote:
| Downsides - unpredictable latency for IO when the parser
| crosses a page boundary and encounters an unbacked page.
| Unpredictable IO issue size (might be smaller IOs than
| ideal). Unpredictable caching behavior (OS may drop the
| pages before the application is done with them, only to
| have to reread again).
|
| You can mitigate this somewhat with explicit MAP_POPULATE
| and mlock.
| Sesse__ wrote:
| Also, generally higher overhead, more memory use for very
| large files (the page tables are not free), and no way of
| handling errors.
| loeg wrote:
| > more memory use for very large files (the page tables
| are not free),
|
| They're... kinda free (in terms of capacity). A PTE or
| PDE is 64 bits per page. Even with 4kB pages, this is
| 8/4096 = 0.2% overhead. Churning the page tables and TLB
| as you walk the file is expensive, or at least
| historically was expensive (used to be that removing an
| entry required flushing the entire TLB; I don't think
| that's true anymore).
|
| If you can use MAP_HUGE_2MB, this drops off to 0.00038%
| and the TLB impact goes way down.
|
| > no way of handling errors.
|
| Yeah, this aspect is really significant. Thanks for
| mentioning it.
| tarasglek wrote:
| The net performance of your system is only as fast as the slowest
| part.
|
| So, you optimized your C IO loop and it's really fast when there
| is not much Python...but soon as you add any Python, you'll be
| bottlenecking in Python and all that fastness wont matter...eg if
| you do some ORM SQL or any other thing that's more complex than
| `return data`
| jeremycarter wrote:
| It's also interesting to compare energy usage. Intel has RAPL
| which allows you to measure the joules of a certain program or
| processor. We should be focusing not just on performance but
| also on environmental runtime impact.
| leni536 wrote:
| Energy efficiency is mostly about finishing faster, so the
| CPU can go to low power state earlier.
| Dachande663 wrote:
| How does yyjson[0] compare to simdjson? Their benchmarks suggest
| it could be a positive.
|
| [0] https://github.com/ibireme/yyjson
| ashvardanian wrote:
| In a nutshell, we use both in UKV. simdjson is faster for read-
| only operations, but it won't help you create a new JSON.
| yyjson is the best library I have seen for creating/updating
| JSONs.
| jammycrisp wrote:
| If you're primarily targeting Python as an application layer,
| you may also want to check out my msgspec library[1]. All the
| perf benefits of e.g. yyjson, but with schema validation like
| pydantic. It regularly benchmarks[2] as the fastest JSON
| library for Python. Much of the overhead of decoding JSON ->
| Python comes from the python layer, and msgspec employs every
| trick I know to minimize that overhead. </sales pitch>
|
| [1]: https://github.com/jcrist/msgspec
|
| [2]: https://github.com/TkTech/json_benchmark
| EVa5I7bHFq9mnYK wrote:
| Haha, 6ms to add two integers that's about 1000x slower than on
| my trusty old PDP-11. You can trust software people to undo half
| a century of hardware engineers efforts).
| jalk wrote:
| That was an end-to-end measurement of the RPC service, so you
| need to measure how long it took the card-reader to read the
| instructions and print out the result on the line printer ;-)
| iddan wrote:
| I'm not sure if FastAPI is the best competition for this library
| as it focuses on building REST APIs rather than JSON RPCs, but
| anyway great work!
| ashvardanian wrote:
| Thank you! You are right, FastAPI doesn't promise performance.
| But on the other hand, they have super usable. We wanted to
| show, that libraries can be both usable and fast, hence the
| name of the post :)
| dan-robertson wrote:
| They say 'high performance' on their homepage:
| https://fastapi.tiangolo.com/
| sixo wrote:
| you would not be crazy to think the "Fast" refers to
| performance
| jeremycarter wrote:
| A common pitfall. FastAPI is orders of magnitude slower
| than Java or .NET. A simple expressjs API outperforms it,
| especially with concurrent users.
| pbreit wrote:
| What is the difference between RPC and API besides the URL &
| payload structure?
| lxe wrote:
| It's just semantics. "managing remote resources" or "calling
| remote procedures" is all just network calls, usually http.
| naasking wrote:
| Semantics are 90% of programming though. Learning syntax is
| simple, learning semantics is hard. Semantics structure how
| you think about problems and express solutions.
| sirsinsalot wrote:
| Semantics is 90% of life!
| pbreit wrote:
| But in this case there is almost no difference. Basically
| same payloads with a slightly different "command" format.
| naasking wrote:
| It's the same payloads with a slightly different command
| format if you structure your program in exactly the same
| way. The point is that you don't structure your program
| in exactly the same way in each case.
| efdee wrote:
| REST is based around resources and kind of the polar opposite
| of RPC when it comes to writing APIs.
| whalesalad wrote:
| REST is literally RPC. You are making a remote procedure
| call to a URL. GET this resource. DELETE this resource.
| It's the same thing.
|
| What is the difference between remote.perform('some-
| action', { payload }) and remote.POST('some-action',
| payload) ?
| pbreit wrote:
| Is it though? Or is the difference mostly semantics? At
| then end of the day it's just barely different ways for
| issuing a "command".
| Genbox wrote:
| simdjson is awesome (Lemire does a lot of great stuff) and
| io_uring is a good use of circular buffers to get efficient IO,
| but I always squirm a little when I see JSON in a high-
| performance context.
| pbreit wrote:
| I was just thinking how lousy a data format JSON is for tabular
| data. Would sending it down in CSV improve things at all?
| loeg wrote:
| No, parsing CSV is also pretty slow. You want some sort of
| length prefixed and ideally fixed width column format.
| ZeroCool2u wrote:
| If anything you'd probably want to send it in Arrow[1] format.
| CSV's don't even preserve data types.
|
| [1]: https://arrow.apache.org/
| pletnes wrote:
| Or parquet, for compression?
| alchemist1e9 wrote:
| arrow/feather is really the best format these days for
| tabular data transmission.
|
| anyone who disagrees I'd be very interested to hear your
| thoughts on alternatives.
| pletnes wrote:
| What about compression - is this part of arrow itself?
| adgjlsfhk1 wrote:
| It's not part of Arrow, but Arrow is columnar so just a
| basic LZ4/ZSTD will work pretty well.
| pbreit wrote:
| Arrow looks super complicated.
|
| Are data types useful for data to/fro web/mobile clients?
| Encode type into the column header?
| adgjlsfhk1 wrote:
| data types are absolutely helpful. when you know a column
| stores Float64 data, you don't have to write out float to
| base 10 and parse it back. You just dump the bytes.
| ashvardanian wrote:
| Yes, we also constantly think about that! In the document
| collections of UKV, for example, we have interoperability
| between JSON, BSON, and MessagePack objects [1]. CSV is another
| potential option, but text-based formats aren't ideal for large
| scale transmissions.
|
| One thing people do - use two protocols. That is the case with
| Apache Arrow Flight RPC = gRPC for tasks, Arrow for data. It is
| a viable path, but compiling gRPC is a nightmare, and we don't
| want to integrate it into our other libraries, as we generally
| compile everything from sources. Seemingly, UJRPC can replace
| gRPC, and for the payload we can continue using Arrow. We will
| see :)
|
| [1]: https://github.com/unum-
| cloud/ukv/blob/main/src/modality_doc...
| sally_glance wrote:
| Are there any synergies with capnproto [1] or is the focus
| here purely on huge payloads?
|
| I'm just an interested hobbyist when it comes to performant
| RPC frameworks but had some fun benchmarking capnproto for a
| small gamedev-related project and it was pretty awesome.
|
| [1] https://capnproto.org/
| ZeroCool2u wrote:
| Though I have read a bit about simdjson I'm not super familiar
| with how it works, so please forgive me if this is a silly
| question.
|
| Since simdjson is where a lot of the performance benefits come
| from here, would you need to compile the underlying C/C++ code on
| the machine that you're deploying on to make sure that simdjson
| is using the correct instruction set? Like what if the processor
| I'm compiling on has AVX-512 support, but the target machine for
| deployment doesn't? Does it just generate the machine code for
| all instruction sets and then can automatically choose to use the
| optimal instructios at runtime? Or does it just have to compile
| every time you pip install the package? I could see this being
| awkward if you're building a docker container perhaps, but maybe
| I'm just woefully uninformed.
| loeg wrote:
| It looks like the vast majority of the benefit (1200us -> 85us)
| comes from using WebSockets instead of REST (HTTP and maybe
| other protocol overhead). 85us beats GRPC in their tests, so
| it's likely adequate for many applications.
| knutzui wrote:
| The README mentions this: > Selects a CPU-tailored parser at
| runtime. No configuration needed.
|
| https://github.com/simdjson/simdjson
| ZeroCool2u wrote:
| Thank you, I missed that when I first looked at the README.
| Very cool if anyone else wants to take a look: https://github
| .com/simdjson/simdjson/blob/master/doc/impleme...
| rpep wrote:
| This isn't a silly question at all, and it's quite complicated.
|
| When compiling C/C++, with _some_ compilers you are able to
| specify multiple architectures and provide a fat binary. It 's
| been a few years now since I've worked on this sort of thing,
| but the Intel compiler for e.g. used to allow you to do
| something like this at the compilation stage
|
| -march=avx,avx2,avx512
|
| And at runtime, your processor would use the most recent
| instruction set. In practice, you don't always want to do this,
| since you produce fat binaries - i.e. every function call
| that's got vector instructions will have multiple versions, and
| the file size goes up, library is slower to load, etc... I
| can't remember if there was an overhead too to each function
| call. So instead of doing this, what is also common is to
| compile N versions of the library (one per instruction set),
| and then load the appropriate version at runtime. This you can
| do with any compiler, and is indeed what Intel do themselves
| with the MKL library - if you look into the package for it, you
| can see that you end up with multiple shared libraries, each
| with a suffix saying which instruction set it supports (e.g.
| libmkl_avx.so, libmkl_avx2.so, libmkl_avx512.so). I'm not
| familiar with simdjson so not sure which approach it takes.
|
| Interestingly, if you use ARM processors, the SVE instruction
| set is designed so that you don't need to re-compile if a new
| processor family comes along with longer vector processing
| units.
| ZeroCool2u wrote:
| Thanks, this is super helpful. I didn't know about that
| feature of the SVE instructions. I was curious and it looks
| like RISC-V takes it a step further even and is "Vector
| Length Agnostic"[1]. Pretty cool!
|
| [1]: https://gms.tf/riscv-vector.html
| anentropic wrote:
| https://www.unum.cloud/ujrpc
|
| Is this supposed to be just a blank page, which is what I see?
| ashvardanian wrote:
| Oh, yes, sorry. Corrected it on the GitHub page. New Doxygen
| documentation portal for all of our open-source libraries is in
| the works. Can't wait to share it :)
| ddorian43 wrote:
| There are also some other blank pages like
| https://www.unum.cloud/ukv/ and
| https://www.unum.cloud/ukv/details
| ashvardanian wrote:
| Yes, same story there :)
| telotortium wrote:
| If you just want to experience the topic of the original post,
| go to https://github.com/unum-cloud/ujrpc.
___________________________________________________________________
(page generated 2023-03-06 23:01 UTC)