[HN Gopher] Show HN: EloqKV - Scalable distributed ACID key-valu...
___________________________________________________________________
Show HN: EloqKV - Scalable distributed ACID key-value database with
Redis API
We're thrilled to unveil EloqKV, a lightning-fast distributed key-
value store with a Redis-compatible API. Built on a new database
architecture called the Data Substrate, EloqKV brings significant
innovations to database design. Here's the unique features that
makes it stand out: - Flexible Deployment: Run it as a single-node
in-memory KV cache, a larger-than-memory database or scale to a
highly available, distributed transactional database with ease. -
High Performance: Achieves performance levels comparable to top in-
memory databases like Redis and DragonflyDB, while significantly
outperforming durable KV stores like KVRocks. - Full ACID
Transactions: Ensures complete transactional integrity, even in
distributed environments. - Independent Resource Scaling: Scale
CPU, memory, storage, and logging resources independently to meet
your needs. We'd love to hear your thoughts and feedback!
Author : hubertzhang
Score : 20 points
Date : 2024-09-19 12:03 UTC (1 days ago)
(HTM) web link (www.eloqdata.com)
(TXT) w3m dump (www.eloqdata.com)
| hodr_super wrote:
| It does seem that there are a lot of activities to
| replace/enhance/improve Redis lately. Dragonfly, Garnet from
| Microsoft, and the BSD licensed Redis fork Valkey, to name just a
| few. Now this.
| hubertzhang wrote:
| Yeah, it's just a happy coincidence that our product release
| lined up with the Redis license change. We think EloqKV is a
| way more powerful and versatile product than Redis. It can
| definitely be used as a simple in-memory cache, just like
| Redis, Valkey, or Dragonfly. But the real strength of EloqKV is
| in its full database capabilities: scalability, consistency,
| and high availability, all the things you'd expect from a
| modern database.
| hubertzhang wrote:
| By the way, I'm Hubert, the CTO of EloqData, I'm the one who
| submitted the post and happy to answer any questions you
| have.
| hodr_super wrote:
| I took a quick look at the evaluation you did on your blog
| section, and the numbers look amazing, a little too good to
| be true to be honest. It must be quite some effort to build
| this, who are you guys?
| PeterZaitsev wrote:
| What is license ? I see Download but it is not clear if it is
| Open Source
| hubertzhang wrote:
| We are not yet open sourcing the code on our github site yet.
| The current release is a prebuilt binary, user can do whatever
| you want with the software (there is a license file in the
| downloaded tarball), EloqKV is not yet production ready, so
| please don't use it in a production environment.
|
| We are a small team of developers, and currently EloqKV is
| still under heavy development. We would like to maintain
| agility and quickly iterate and improve our product with
| consistency with our customer's feedback. However, we are
| actively evaluating multiple paths to open sourcing our
| technology and any suggestions and concerns will certainly be
| kept in our mind as we progress.
| PeterZaitsev wrote:
| Hi,
|
| Having Open Source Project on GitHub does not mean you need
| to become less agile. First there will not be instantly 1000
| of people sending you pull requests, second even if they do -
| you have no obligation to take them in.
|
| However if you say, we can write crap code... and no one
| would know because they only see binaries, this is not really
| a way gain a good confidence
|
| In my opinion those days just having a "binary you can use"
| is the worst way to go for something like database/data
| store. If you are not sure about your Open Source plans
| having something like SaaS solution with free tier so users
| can experiment easily could be way to go
| hubertzhang wrote:
| Thank you for the suggestions. We are currently developing
| a cloud-native version of EloqKV, with both Serverless and
| BYOC (Bring Your Own Cloud) options underway. The binary is
| a technology preview for those interested in experiencing
| the new architecture. EloqKV is built with a cloud-native
| focus, featuring a quaternary decoupling architecture that
| enables independent scaling of CPU, memory, storage, and
| logging. This design maximizes cloud elasticity for optimal
| scalability and resource efficiency. So I completely agree
| with the suggestion to offer a SaaS solution, allowing
| users to experiment.
| hodr_super wrote:
| If it's not open-sourced, I'm not going to use it to store my
| critical data. Just saying...
| ljchen wrote:
| I second this. It's too risky and simply not worth it.
| Nevertheless, I find this "optional ACID" thing interesting.
| Many years ago when I was a graduate student, NoSQL was a big
| thing. It was widely claimed that transactions were expensive
| and you had to drop them in exchange to scale. I always had
| this question that if transactions were the culprit, why not
| turning them off? I later found that the relational system is
| such a monolith that everything (caching, concurrency
| control, logging, locking) is wired together in an extremely
| complex way and there is simply no "turning off".
| throwaway81523 wrote:
| Redis simply serializes every operation, I thought.
| Transaction = run a Lua script as a single operation. I
| think that is ACID, if you count RAM as "durable" and doing
| one thing at a time as "concurrent".
| the_precipitate wrote:
| Playing devil's advocate here, I do have some doubts about the
| future dynamics between open source and database systems. Andy
| Pavlo mentioned in his CMU DB course that nowadays new database
| companies are either fully closed-source or only open a small
| portion of non-essential code.
|
| If you think about it, databases are arguably a more critical
| piece of IT infrastructure than cybersecurity products. Yet,
| while cybersecurity companies are thriving with many company
| valuations exceeding $10 billion, database companies--
| especially those embracing open source--struggle to achieve
| similar commercial success. The few database companies that do
| reach high valuations are typically based on closed-source
| products, while those adopting open-source models often face
| significant challenges in becoming profitable.
|
| It's also worth noting that many companies in the database
| space have recently changed their licenses around open source
| or source availability. If major players like MongoDB,
| ElasticSearch, and Redis are all making these tough decisions
| to build sustainable businesses, it might not be fair to simply
| blame corporate greed. Without adequate returns on investment,
| venture funding for database development could dry up, which
| would ultimately stifle innovation in this crucial sector.
| the_precipitate wrote:
| This is really interesting! The performance looks impressive.
| I've always struggled with the Redis/MySQL two-tier architecture
| because they are two completely different systems. Porting an
| application is always tricky due to API incompatibility, and
| maintaining consistency between the two is a huge hassle. If
| there's any issue on the cache side (like performance jitters or
| a few servers going down), the DBMS part can quickly get
| overwhelmed. This kind of cascading failure is a common problem
| in distributed systems. There are some great discussions on this
| topic over at https://danluu.com/cache-incidents/. BTW, although
| the formatting is a bit rough, it's a great read for anyone
| interested in caching. Back to the topic, I always hoped that new
| advancements in database systems can alleviate or eliminate this
| problem, your work seems to be on the right track.
| hubertzhang wrote:
| Thank you for your interest. I agree--if Redis (as a cache) and
| MySQL (as a store) are deployed separately, we can only achieve
| eventual consistency. That's precisely why we're integrating
| cache and storage into a single system, much like EloqKV.
| apavlo wrote:
| > EloqKV brings significant innovations to database design
|
| What is the novel part? I read your "Introduction to Data
| Substrate" blog article and the architecture you are describing
| sounds like NuoDB from the early 2010s. The only difference is
| that NuoDB scales out the in-memory cache by adding more of what
| they call "Transaction Engine" nodes whereas you are scaling up
| the "TxMap" node?
|
| See also Viktor Leis' CIDR 2023 paper with the Great Phil
| Bernstein:
|
| * https://www.cidrdb.org/cidr2023/papers/p50-ziegler.pdf
|
| * https://youtu.be/tiMvcqIfWyA
| eloqdata wrote:
| Thank you, Andy! This is Jeff, CEO and Chief Architect of
| EloqData. It's a great honor for us to have THE Andy Pavlo join
| the discussion on our first HN submission.
|
| If I remember correctly, NuoDB uses a shared cache with a cache
| coherence protocol, whereas EloqKV uses a shared nothing
| (partitioned) cache. The former is a local read but needs to
| broadcast each write to all nodes. The latter has no broadcast
| for writes but may be a remote read. The tradeoff is evident
| and we are actively exploring opportunities to strike a
| balance, e.g., for frequently-read, rarely-write data items,
| use the shared cache mode.
|
| We appreciate you pointing us to the CIDR paper. I had the
| pleasure of working with Phil for some time and fondly remember
| many discussions with Phil on various topics many years ago. To
| address your question, yes, we've been trying to solve the
| research challenges presented in the CIDR paper. The devil is
| in the details. We've developed numerous new algorithms and
| invested significant engineering effort into the design and
| implementation of our products. The benefits are as follows:
|
| - Optimality: We believe we have an overall design that
| optimizes synchronous disk writes and network round-trips. For
| instance, when the design is reduced to a single node, its
| performance matches or exceeds that of single-node servers. As
| you might expect, a lot of innovation has gone into making
| distributed transactions as efficient as non-distributed ones,
| comparable to those in MySQL or PostgreSQL.
|
| - Modularity: Our architecture allows us to easily replace the
| Parser/Compute layer and Storage/Persistence layer with the
| best existing solutions. This means we can create new databases
| by leveraging existing parsers and compute engines from current
| database implementations to achieve API-compatibility, as well
| as leveraging existing high-performance KV stores for the
| persistence layer. This allows us to avoid reinventing the
| wheels and to take advantage of decades of innovations in the
| database community.
|
| - Scalability: The entire system operates without a single
| synchronization point--not even a global sequencer. We drew
| many inspirations from the Hekaton and your TicToc paper. All
| four types of resources (CPU, Memory, Storage, Logging) can be
| scaled independently, as we mentioned earlier. More
| importantly, they can scale dynamically to accommodate workload
| changes without service disruptions.
|
| We look forward to sharing more technical details as we move
| out of stealth mode. I hope to continue this conversation with
| you in person in the near future.
| hubertzhang wrote:
| Note that this binary release is a preview of EloqKV. We're
| actively developing a cloud-native version with seamless
| scalability and a serverless experience. Stay tuned--it's coming
| soon!
| hubertzhang wrote:
| This binary release is a preview of EloqKV. We're actively
| developing a cloud-native version with seamless scalability and a
| serverless experience. Stay tuned--it's coming soon!
| marcodiego wrote:
| License?
| jacobn wrote:
| Jepsen test?
| henning wrote:
| Show us the Jepsen tests. Hire Aphyr to test your database.
| fizx wrote:
| Redis's transaction api is terrible, and doesn't work across
| shards. Any reasonable transactions are done in Lua, and because
| Lua mostly works well, there's not a lot of pressure to fix
| transactions.
|
| However, if you're giving redis access to different tenants, Lua
| is too dangerous.
|
| I'd love to see a "real" transaction API for Redis.
___________________________________________________________________
(page generated 2024-09-20 23:01 UTC)