[HN Gopher] Redis vs. KeyDB vs. Dragonfly vs. Skytable
___________________________________________________________________
Redis vs. KeyDB vs. Dragonfly vs. Skytable
It seems that several projects claim that they have built the
"world's fastest key/value store" or sometimes the phrase used is
even more outrageous. The following projects are in question: -
Redis: https://github.com/redis/redis - KeyDB:
https://github.com/snapchat/keydb - Dragonfly:
https://github.com/dragonflydb/dragonfly - Skytable:
https://github.com/skytable/skytable And, these are my benchmark
results (30 threads, 1 client per thread with GET/SET. See below
for setup): 1. Redis: 112,100 / 99,892 2. KeyDB: 288,931 /
282,997 3. Dragonfly: 408,322 / 392,446 4. Skytable: 619,992 /
676,091 # Notes 1. Redis: I'll start with Redis which I'd like to
call the "original" key/value store (after memcached) because it is
the oldest and most widely used of all. Being a long-time follower
of Redis, I do know it's single-threaded (and uses io-threads since
6.0) and hence it achieves lesser throughput than the other stores
listed above which are multi-threaded, at least to some extent. The
best parts about Redis: it's the most feature complete of all the
systems here, and is the oldest. (not saying old necessarily means
better). 2. KeyDB: The second is KeyDB. IIRC, I saw it in a blog
post which said that it is a "multithreaded fork of Redis that is
5X faster"[1]. I really liked the idea because I was previously
running several instances of Redis on the same node and proxying
them like a "single-node cluster." Why? To increase CPU
utilization. A single KeyDB instance could replace the unwanted
proxying funkiness, so I ditched Redis for KeyDB. Has been a fine
experience so far, except for some occasional crashes. 3.
Dragonfly: Just found it on HN and seems to be relatively new.
Dragonfly claims that it is 25X faster (that I couldn't reproduce)
than Redis[2] and has the slogan "Probably, the fastest in-memory
store in the universe!". Doesn't support all the Redis commands
yet, but I find it interesting mainly because of performance. Also,
it's good to know why it is faster because it clearly outlines the
underlying architecture[2]. The other three stores don't say much
about it. Will be following the project. 4. Skytable: Found it
while looking for projects written in Rust. Claims to be "insanely
fast." Skytable's "experimental benchmarks" claim that it is
something around 10X faster than Redis and some 2X-3X faster than
KeyDB[3]. I hadn't heard of Skytable and it doesn't seem to be as
widely used (unless I'm missing something?). I find it interesting
because of the planned features[4] and performance. Only Skytable
natively runs on Windows out of the four. Will be following the
project. 5. My thoughts: Redis needs no introduction and is
arguably super stable for use on production systems (using widely
in our systems). KeyDB seems to be "stable enough" and it seems to
be good for prod since Snapchat uses it already[5] (and so do we!).
I found no Dragonfly v/s Skytable benchmarks. The best part about
Redis, KeyDB and Skytable is that they don't make any "crazy
assumptions" about the system they run on. What do I mean?
Dragonfly expects you to have the latest hardware[6] and the latest
kernel[7] on all your servers. I find this outrageous because not
all servers run 5.10 and a majority of them are still using long-
running branches on 4.x. Also, I don't expect them to have the
latest processors, either. I'd argue if the other three stores
started assuming the latest features, they'd be far faster than
what they are today. Finally, both Dragonfly and Skytable are still
early in their development so it may not be fair to compare their
features against Redis and KeyDB who have been around for far
longer. Also, all projects other than Skytable have companies
behind them (unless I'm missing something). Edit #1: I have run
the benchmarks myself for each store. In the benchmark with Redis,
KeyDB and Skytable (redis-benchmark, memtier and sky-bench): Redis
and KeyDB benchmarks seem to be very consistent, Skytable is a
little inconsistent at times. However, in the benchmark with Redis,
KeyDB and Dragonfly (with memtier), I was NOT able to reproduce the
25X speed that Dragonfly claims. I ran all tests on two m5.4xlarge
servers (one with the k/v store and one with the benchmark tool).
Edit #2: Added benchmark results What are your thoughts? Have you
tried benchmarking any of them locally or in the cloud?
References: [1]: https://docs.keydb.dev/blog/2019/10/07/blog-post
[2]: https://github.com/dragonflydb/dragonfly [3]:
https://github.com/ohsayan/sky-benches [4]:
https://github.com/skytable/skytable/issues/203 [5]:
https://docs.keydb.dev/news/2022/05/12/keydb-joins-snap [6]:
https://github.com/dragonflydb/dragonfly/issues/124 [7]:
https://github.com/dragonflydb/dragonfly/issues/96
Author : johnwoods
Score : 112 points
Date : 2022-06-19 05:05 UTC (17 hours ago)
| TheChaplain wrote:
| I'm trying to get my workplace to accept Redis for simple caching
| but they're not open to the idea. Instead they're all in
| Hazelcast but real stingy about who gets to use it and for how
| much memory.
|
| I just want to cache some objects and avoid hammer the DB on
| every request :'(
| ghusbands wrote:
| You could probably find a local caching library to do the same
| thing in-process (via mmap) and partially reduce workload that
| way. Hopefully, using a local library needs less managerial
| involvement than installing/using a service.
| FridgeSeal wrote:
| > Dragonfly expects you to have the latest hardware[6] and the
| latest kernel[7] on all your servers. I find this outrageous
| because not all servers run 5.10
|
| I don't understand why this is "outrageous"? There's performance
| features and better hardware utilisation in newer kernels? If you
| _want_ the performance offered by the applications built on top
| of these, then arguably you should be prepared to run the
| hardware and software required to achieve this.
| johnwoods wrote:
| not if I can get equivalent (if not better) performance on my
| existing hardware
| FridgeSeal wrote:
| From your own numbers, Dragonfly get ~3.6-4 times better perf
| than Redis, and better than keydb (which is "just"
| multhreaded redis-simplifying). That indicates pretty clearly
| to me that we're not getting same-or even close performance-
| on existing hardware; and ultimately, if performance is that
| important, shouldn't we be giving our applications every
| advantage they can get?
| viraptor wrote:
| > Dragonfly expects you to have the latest hardware[6] and the
| latest kernel[7] on all your servers. I find this outrageous
| because not all servers run 5.10 and a majority of them are still
| using long-running branches on 4.x.
|
| I don't see anything wrong with that. Either your kv store is
| performing OK or can be scaled using the usual methods and it's
| not holding you back... or you actually critically depend on the
| fastest kv performance you can get. In the second case, you'll be
| likely happy to dedicate some time to standing up a couple of
| machines with updated kernels.
| tyingq wrote:
| As far as I can tell, "latest kernel" really means "has
| io_uring"...which is core to how Dragonfly works, so it seems
| reasonable to me.
| cperciva wrote:
| _I have run the benchmarks myself for each store._
|
| I feel like I'm missing something here. Which benchmarks did you
| run, and what were the results?
| johnwoods wrote:
| updated the post. I ran the benches that were mentioned here[1]
| and here[2].
|
| [1]: https://github.com/dragonflydb/dragonfly#benchmarks [2]:
| https://github.com/ohsayan/sky-benches
| websap wrote:
| If you're running a really old version of Linux, you're probably
| giving up on many-many performance optimizations.
|
| > I find this outrageous because not all servers run 5.10 and a
| majority of them are still using long-running branches on 4.x
|
| I don't think it's outrageous to ask people to upgrade 5.10. This
| was released over 1.5 year ago and there's no reason why your
| corporate overload shouldn't consider migrating to this, or at
| least have someone working on qualifying their workloads.
| qatanah wrote:
| Also if you can add amazon memory db to yor list of benchmarks,
| would also be interested to know more.
|
| https://aws.amazon.com/memorydb/
| theflyinghorse wrote:
| Is it not just Redis that AWS offers as a managed service?
| ksec wrote:
| >Redis, which I'd like to call the "original" key/value store
|
| My old friend Memcached just came by and want to say Hello. He is
| still alive and kicking. Netflix has been treating him well.
| lazyjones wrote:
| Memcached isn't even a "key/value" store (in database terms),
| it's a cache.
|
| https://github.com/memcached/memcached/wiki/ProgrammingFAQ#w...
| scns wrote:
| There is a 100% binary protocol compatible implementation in
| Rust.
|
| https://memc.rs/intro
| [deleted]
| lapser wrote:
| Wow! I was under the very wrong impression that memcached
| hadn't been updated in a long time. It has been having regular
| releases for longer than I thought. Great to hear!
| ksec wrote:
| Somewhere along the ling, lots of ( well deserved ) hype went
| to Redis and Postgres, and people thought Memcached and MySQL
| died.
|
| I have been wondering for a long time if we will ever see
| Memcached 2.0 and MySQL 9.0
| zeckalpha wrote:
| It has cache in the name so people are less likely to use it
| inappropriately.
| johnwoods wrote:
| > My old friend Memcached just came by and want to say Hello.
| He is still alive and kicking. Netflix has been treating him
| well.
|
| Forgot my dear buddy. Added it back.
| inferiorhuman wrote:
| BerkeleyDB's been around since the mid-90s...
| forty wrote:
| One important thing I would look at is "is there independent
| managed hosting options", in case at some point you have better
| things to do than managing database and having to care if it's
| the right hardware and kernel version. I know at least redis
| shines on this, with managed offering on all major cloud
| provider.
| cientifico wrote:
| Why do you consider single threaded as a reason to have less
| throughput?
|
| Are you looking for throughput or speed?
|
| Also, what are your needs?
| codetiger wrote:
| He has mentioned that to gain better CPU usage he runs multiple
| redis on the same machine. Probably the only issue is unused
| CPU
| spaintech wrote:
| This article interests me as it has been a bucket list todo to
| get a good sense of some of the key/value offerings, thank you
| for posting them.
|
| Quick ask: I don't see "some" of the other offering out there
| like MemCached... what was the criteria used to select these? I
| don't see any source of how the test where run, specs of the
| systems, how the DB where set up, etc. Would be very valuable to
| have in order to attempt to re-validate these test on our own
| platform. I also came back and saw some of your updates to the
| outcomes, While the Get/SET numbers are provided, I don't find a
| reference to time, resource utilization, or time which would IMO
| make the data more valuable in understating how the numbers where
| achieved. * I might be a factor of the benchmarks themselves, but
| would add value IMO to your fantastic post.
| fmajid wrote:
| KeyDB also has another key advantage over Redis: if your data set
| exceeds RAM, it will spill to disk (SSD) and maintain good
| performance.
| romange wrote:
| Hi, I am the author of Dragonfly. Following this post, I
| performed the loadtest on instance m5.4xlarge, recorded myself
| and posted the video here:
| http://assets.dragonflydb.io/videos/video2609676488.mp4
|
| the command I used to load was: "memtier_benchmark --ratio 1:0 -t
| 28 -c 20 -n 100000 --hide-histogram --distinct-client-seed -d
| 256"
|
| I did the recording to clear out any doubts whether my benchmark
| results are real.
| w-ll wrote:
| Have YOU tried benchmarking any of them locally or in the cloud?
| johnwoods wrote:
| updated the post
| w-ll wrote:
| I use Redis because of some of the extra features for sets
| and queues and such.
|
| If I just wanted pure kv store i'd still go memcache. local
| install for stuff that can expire often (html template
| caches) and reach out to a network store for user sessions
| and the like.
|
| all my kv caches are designed to be rebuilt and not care. if
| you need ACID kv go with a proper db.
| didip wrote:
| Anyone knows where can I find a good helm chart for KeyDB?
| throwaway81523 wrote:
| How about including Pedis in this? It is a Redis partial
| reimplementation, written around Seastar:
| https://github.com/fastio/1store
| judofyr wrote:
| > I was NOT able to reproduce the 25X speed that Dragonfly claims
|
| The Dragonfly benchmark which showed 25x faster was run on
| c6gn.16xlarge which has 64 vCPU. The m5.4xlarge you used "only"
| has 16 vCPU. Since Dragonfly is achieving its performance by
| using multiple threads it's not weird that you weren't able to
| replicate its performance gain.
|
| In my opinion, when it comes to these types of multi-threaded
| benchmarks it's much better to separate the "baseline, one-
| process performance" from "how it scales with number of
| processes". E.g. if you first pin Dragonfly to only run on a
| single core you can find the baseline performance compared to
| Redis, and then you can run different benchmarks of Dragonfly
| with increasing number of cores to see how it scales. This should
| give you much better understanding of how the system behaves.
| (The release of Dragonfly actually prompted me to write a blog
| post on this: https://t.holmium.no/dia/apples-vs-oranges/)
| johnwoods wrote:
| agree with your article entirely. the dragonfly benchmark seems
| far too artificial to be true and used in practice (given the
| expensive hardware). my tests were on a relatively cheaper box,
| and Skytable topped all the other systems.
| ghusbands wrote:
| If you start with "baseline, one-process performance", you're
| likely to mislead yourself, since no modern server has fewer
| than eight, and a single-thread benchmark can omit important
| locks. A modern baseline is probably two to four cores, even if
| you're running multiple services on one machine.
| vlovich123 wrote:
| I think you misunderstand. You start with 1 then test 2, 4,
| 8... up to the largest core count you can. This shows you
| where the multi threading arch breaks down.
| bradknowles wrote:
| Sorry, you say you ran the benchmarks, but where in this post do
| you link to your results?
| johnwoods wrote:
| updated the post with the overall result for now
| bradknowles wrote:
| Thanks! Much appreciated!
| dmitriid wrote:
| Redis and sqlite are two projects I can't really believe exist.
| They both are absolutely magical.
| Cwizard wrote:
| Could you explain why?
| dmitriid wrote:
| Both are small projects by small teams (though I don't know
| how small Redis team is now). They have insane performance
| characteristics with rather small footprints _and_ a wealth
| of functionality.
| blagie wrote:
| My experience is that small, dedicated teams of high-
| performing individuals are the only way to build tools like
| these. The level of architectural and design coherence
| starts to decrease at 5 people, and falls like a rock
| beyond 10-20.
|
| The key problem is building a firewall so that such people
| can focus. The *organizational* problems there are hard. If
| you have a firewall around five people, you can't tell if:
|
| - They're sitting at home drinking beer and watching cat
| videos
|
| - Are developing the world's fastest database
|
| Which leads to meetings, presentations, funding proposals,
| and so on. Which, in turn, leads to the need for large
| teams.
| cmacleod4 wrote:
| .. and both happen to have strong links to Tcl :-)
___________________________________________________________________
(page generated 2022-06-19 23:02 UTC)