[HN Gopher] Show HN: EloqDoc: MongoDB-compatible doc DB with obj...
___________________________________________________________________
Show HN: EloqDoc: MongoDB-compatible doc DB with object storage as
first citizen
We're excited to share EloqDoc, a new open source document database
built on top of Data Substrate. EloqDoc is designed around the
principle of treating object storage (like S3) as a first-class
citizen for durability and cost efficiency. If you love the
flexibility of MongoDB's document model but are struggling with
scaling, cost, and consistency due to its coupled architecture,
EloqDoc is for you. It's built to solve MongoDB's inherent
infrastructure challenges while remaining fully compatible with
existing MongoDB clients and drivers. Key Features: 1. Object
Storage as First Citizen: Uses object storage for primary
durability, leveraging local NVMe caching to achieve both lower
cost and higher performance than using block-level storage (e.g.
EBS). 2. Decoupled Compute & Storage: Scale your compute/QPS
independently of your storage capacity, or vice-versa, without data
movement. 3. True ACID Transactions: Delivers full ACID compliance
with especially fast distributed transactions--consistency without
compromise. 4. Native Distribution & Multi-Writer: It's a natively
distributed database, eliminating complex manual sharding routers
(like mongos) and supporting true Multi-Writer scalability. Check
it out: https://www.github.com/eloqdata/eloqdoc We welcome any
feedback, critique, or questions on the EloqDoc!
Author : iamlintaoz
Score : 26 points
Date : 2025-10-19 14:57 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| fluxkernel wrote:
| There is a discussion on Reddit for a 5TB data volume but very
| low throughput, I think Object Storage based document database
| would help in this case.
| https://www.reddit.com/r/mongodb/comments/1o41dta/is_there_a...
| hodr_super wrote:
| Mongo Atlas also supports using archive database to store
| infrequently accessed cold data. It's also object storage
| based.
| fluxkernel wrote:
| Typically a separate archive database is not desired choice.
| You need to specify the rule how old the data is move from
| production database to archive database. Also the archive
| database is read only and slow, you can not achieve
| transaction between production database and archive database.
| iamlintaoz wrote:
| Yes, this is exactly the typical use case where EloqDoc shines.
|
| Our auto-tiering feature allows you to manage a massive
| database (e.g., 5TB+) primarily on cheap, durable object
| storage (like S3), which is already built for cross-AZ
| replication and is up to 3x cheaper than traditional cloud
| block storage (EBS).
|
| We use local NVMe SSDs (200K+ IOPS) purely as a high-
| performance cache layer to accelerate read access. Hot and cold
| data are automatically swapped across memory, SSD, and object
| storage tiers based on access frequency, ensuring high
| performance (up to 100x faster than archive solutions)
| the_precipitate wrote:
| FerretDB and DocumentDB seem to be the only other two Mongo-
| compatible open source database implementations. Both are based
| on PostgreSQL.
| iamlintaoz wrote:
| That's correct. FerretDB and DocumentDB both pursue PostgreSQL-
| based document models.
|
| EloqDoc follows a fundamentally different architectural path.
| Instead of executing in Postgres, we leverage the existing
| MongoDB parser and executor to ensure maximum compatibility.
| Our main contribution is to replace the single-writer
| WiredTiger storage engine entirely with Data Substrate.
|
| The Data Substrate is an abstract layer built specifically to
| handle distributed database fundamentals: scalable buffer
| pooling, concurrency control, durability, elasticity, and fault
| tolerance. You can read more:
| https://www.eloqdata.com/blog/2025/07/14/technology
|
| This architectural choice is what enables EloqDoc to deliver
| features that Postgres-based solutions cannot easily match,
| including: full compatibility, native Multi-Writer capability,
| Object Storage First and extremely low-latency distributed
| transactions.
| matharmin wrote:
| FoundationDB also has a Mongo-compatible document layer, but it
| seems like the last release was 6 years ago, so probably
| doesn't count anymore.
| PeterZaitsev wrote:
| I see the license is AGPLv3 while current MongoDB versions as
| SSPL - does it means you took the MongoDB Parser circa 2018 as a
| base ?
| iamlintaoz wrote:
| That is right. We leverage some of the AGPL MongoDB code for
| the parser, as indicated by the License. Our own code can be
| licensed differently, see a previous discussion on Hacker News
| [1]. The Mongo API are reasonably stable over the last few
| years, only seeing very minor changes. Most of the later
| versions improved upon performance and transactions, which we
| support natively with our underlying technologies. Still, if
| you have any specific API that you feel is needed, we'd be
| happy to implement and we welcome community contributions.
|
| [1] https://news.ycombinator.com/item?id=44937978
| freakynit wrote:
| Nice one..
|
| Btw, your login/signup page auth is broken. Isn't moving ahead
| after google auth. Fyi, I use firefox in strict privacy mode.
| iamlintaoz wrote:
| Thank you so much for reporting this! My local testing on
| Chrome privacy mode was fine, but we will definitely prioritize
| checking and fixing the issue in Firefox's Strict Privacy Mode.
| freakynit wrote:
| Cool.. thanks..
| iamlintaoz wrote:
| For full transparency, I am the CEO of EloqData and I am happy to
| answer any questions you have. EloqDoc has been open-sourced for
| awhile, and we are glad to announce that the current code is
| stable for you to test on your workload.
| matharmin wrote:
| The project looks great! Object storage is often so much better
| in terms of cost efficiency than a database on EBS. It's often
| 10-20x more expensive for EBS after taking into account that you
| need 3x replicas for a typical MongoDB deployment, and need to
| over-provision the storage. And being able to scale compute
| independently from storage is great.
|
| The biggest things I'm missing from the docs (checked the github
| page and the site) is seeing what MongoDB features are supported
| or not. I've worked with Azure CosmosDB before, and even though
| it claims MongoDB compatibility, it has many compatibility issues
| as soon as you have more than a basic CRUD application. Some
| examples include proper ChangeStream support, partial index
| support, multi-key index support, set of supported aggregation
| pipeline operations, tailable cursor support, snapshot queries.
|
| Another thing that's not clear: What does multi-master/multi-
| write mean in practice? What happens if you write to the same
| data at the same time on different nodes?
| iamlintaoz wrote:
| That's exactly the reason. S3 is better in almost all aspects
| compared with EBS, except the performance part, and I am glad
| that our Data Substrate technology solved this issue gracefully
| [1].
|
| As for the compatibility, we are leveraging some of the code
| from 4.03 version (the last AGPL version), and we have a very
| good compatibility (we will show some results in later blog
| posts). As I mentioned in another reply post, the Mongo APIs
| are reasonably stable over the last few years, only seeing very
| minor changes. Most of the later versions improved upon
| performance and transaction supports, which we support natively
| with our underlying data substrate technologies. Still, if you
| have any specific API that you feel is needed, we'd be happy to
| implement and we welcome community contributions.
|
| Multi-master/multi-writer means it is a fully distributed
| database. Of course you can run it in single node
| configurations and get all the single node benefits, but if
| deployed in a cluster, you do not need to worry about which
| node to write to, or how data are sharded. If you writes
| potentially can cause conflicts (i.e. write to the same data at
| the same time on different nodes), the concurrency-control will
| handle that for you. In fact, you will encounter the same issue
| even in a single node configuration, since a single node is
| still multi-threaded.
|
| [1] https://www.eloqdata.com/blog/2025/07/16/data-substrate-
| bene...
| akshayshah wrote:
| Very cool! Using "object storage for primary durability" seems
| difficult for any OLTP workload that's latency-sensitive -
| there's a fundamental tradeoff between larger batch sizes to
| control write costs and smaller batches to reduce latency. This
| hurts OLTP workloads especially badly because applications often
| make multiple small writes to serve a single user-facing request.
| How does EloqKV navigate this tradeoff?
|
| Also, I'd love to see:
|
| - A benchmark that digs into latency, throughput, and cost for a
| single workload. Most of the benchmarks I saw are throughput-
| only.
|
| - Some explanation of the "patented 1PC protocol." Your website
| [1] suggests that you treat single EBS volumes as high-
| durability, replicated storage, which seems unusual to me - apart
| from io2 volumes, EBS is designed for less than 3 nines of
| durability [2].
|
| [1]: http://www.eloqdata.com/blog/2025/07/15/data-substrate-
| detai...
|
| [2]: https://aws.amazon.com/ebs/features/
| iamlintaoz wrote:
| These are great questions. I appreciate you carefully reading
| through the documents. For the first question, we have detailed
| benchmark on EloqKV, with the same architecture (but with Redis
| API) in our blog, and we will soon publish more about the
| performance characteristics of EloqDoc. Overall, we achieve
| about the same performance as using local NVME SSD, even when
| we use S3 as the primary storage, and the performance often
| exceed the original database implementation (in the case of
| EloqDoc, original MongoDB).
|
| As for the durability part, our key innovations is to split
| state into 3 parts: in memory, in WAL, and in data storage. We
| use a small EBS volume for WAL, and storage is in S3. So,
| durability is guaranteed by [Storage AND (WAL OR Mem)). Unless
| Storage (S3) fails, or Both WAL (i.e. EBS lost) AND Mem fail
| (i.e. node crash), persistence is guaranteed. You can see the
| explanation in [1]
|
| [1] https://www.eloqdata.com/blog/2025/07/16/data-substrate-
| bene...
| lioeters wrote:
| Oh how interesting, there are related database projects built on
| the same foundation library/tech called Data Substrate.
|
| - EloqSQL - MySQL-compatible, high performance, elastic,
| distributed SQL database https://github.com/eloqdata/eloqsql
|
| - EloqKV - Redis/Valkey Compatible Distributed Transactional Key-
| Value Store https://github.com/eloqdata/eloqkv
___________________________________________________________________
(page generated 2025-10-20 23:01 UTC)