[HN Gopher] Manticore 6.0.0 - a faster alternative to Elasticsea...
___________________________________________________________________
Manticore 6.0.0 - a faster alternative to Elasticsearch in C++
Author : snikolaev
Score : 135 points
Date : 2023-02-10 15:37 UTC (7 hours ago)
(HTM) web link (manticoresearch.com)
(TXT) w3m dump (manticoresearch.com)
| trallnag wrote:
| Time to rewrite it in Rust
| boyter wrote:
| I highly recommend this software if its capabilities fall into
| what you need. It is very fast both in terms of indexing speed
| and search. It's relatively simple to setup and start working
| against, and I have found it very reliable.
|
| It works best when you have a SQL data store you want to index
| against, but with the real time index you can treat it more like
| elastic and other searches. However for that first use case of
| SQL, I don't know of anything else that comes close to being as
| easy to use.
|
| Simply point it at your database, give it a query to pull what
| you want to index and you are done. I suspect that this covers
| about 90% of use cases out there.
|
| If you need more than what the DB native indexing is giving you
| give it a try.
| riku_iki wrote:
| Does it have distributed partitioning like es?..
| snikolaev wrote:
| Yes it does
| https://manual.manticoresearch.com/Searching/Distributed_sea...
| nathanwh wrote:
| From the doc linked "Partitioning is done manually.", which
| seems like a pretty big limitation to me? For static data
| sure, but for non static data it seems like it would be an
| operational headache to manage manual partitions, compared to
| elasticsearch which partitions for you (although IIRC shard
| count is fixed)
| snikolaev wrote:
| Despite years of users relying on distributed indexes and
| manual sharding in Manticore, I concur that automation
| would simplify the process greatly. The development team is
| working on automating sharding and orchestrating the
| shards. Hopefully it will be included in the next major
| release in a few months.
| quijoteuniv wrote:
| What is people looking for in alternatives for elastic search? I
| have been toying in docker with a version of elastic search,
| fscrawler and workplace search to get a company to have better
| access to their knowdledge base. They have exchange, manuals,
| emails,images& video github and other stuff... does this
| alternatives have connectors too? Any experience on this?
| snikolaev wrote:
| There are several reasons why some people prefer alternatives
| to Elasticsearch, including:
|
| * License preference: Some people prefer true open-source
| licenses as opposed to the license that Elasticsearch has
| switched to.
|
| * Performance and resource consumption: For some, performance
| and resource consumption are significant factors in their
| choice of a search engine.
|
| * SQL vs JSON DSL: Some people prefer using SQL over
| Elasticsearch's JSON domain-specific language.
|
| * Maintenance: Some believe that maintaining Elasticsearch can
| become challenging when the data collection becomes large
| enough.
|
| That's what I've heard from those who preferred Manticore over
| Elasticsearch.
| vosper wrote:
| > License preference: Some people prefer true open-source
| licenses as opposed to the license that Elasticsearch has
| switched to.
|
| Those people can also use OpenSearch, which is a recent fork
| of ElasticSearch (by Amazon) that is using the Apache 2.0
| license.
| rzzzt wrote:
| "We are excited to announce the addition of telemetry in this
| release. [...] This feature can be easily turned off in the
| settings if desired."
|
| How did they know in the previous 5 major versions which part of
| the product to improve?
| snikolaev wrote:
| We didn't, hence we may have inadvertently improved the wrong
| areas. Although we have been receiving feedback from the
| community, it was never backed by concrete data and thus, we
| had to rely on our best guess.
| reilly3000 wrote:
| This is the best appeal for OSS I've seen in a while.
|
| Actively telling a project how to improve vs passively
| observing how the system is used in the aggregate: that is a
| choice each person should be offered to engage with the
| project, along with the right to lurk in peace or fix their
| problems their own ways.
|
| The data itself ought to be public, and if it can't be then
| it shouldn't be gathered. The insights from that data should
| come from the community, not just project leads. That data's
| relevance to product development and design should be
| annotated on feature tickets.
|
| I am good with having a relationship with a project that
| involves sharing my behaviors within the system, others are
| not. As a user I do I want to be able to see if I'm way off
| the beaten path or using a popular method when I'm deciding
| if I should debug, research, or work around a problem. As a
| maintainer team, your time is precious and the BS is already
| thick, so I feel providing that visibility is the least I can
| do to contribute to your work.
|
| I feel very differently about this with respect to for-
| profits and my PII tied to behavioral records.
| mooreds wrote:
| I looked for an up to date elasticsearch compatibility chart but
| was unable to find one.
|
| I found an article from 2022 that did a compare/contrast but I
| wanted a feature by feature breakdown.
| juxtaposicion wrote:
| How does this compare to Quickwit or other Tantivy-powered
| engines?
| remram wrote:
| I think quickwit is more tailored towards querying large
| indexes sitting on S3 than fast queries from a local or in-
| memory index.
|
| lnx might be similar, I'm not sure. It's very new and I had a
| bad experience trying it out.
| caseyf wrote:
| I've been running Manticore (previously SphinxSearch) on a
| faceted search heavy site with a million MAUs for 15 years. I'd
| definitely use it again for another project.
|
| If the data that you want to search is entirely contained in a
| SQL database, it's an uncomplicated and powerful solution,
| definitely check it out. If not, Manticore may still be a nice
| solution for you, but I can't speak to that.
| Multrex wrote:
| Any chance to use it with Graylog instead of Elasticsearch or
| Opensearch?
| entropyie wrote:
| See also this lightweight alternative to ES:
| https://github.com/zinclabs/zinc
| remram wrote:
| That looks great. I've been needing a multi-platform low-
| resource full text search engine/library.
| rbanffy wrote:
| I think I may give it a shot at my home network's ELK stack.
| Elastic is a memory hog.
| comrad wrote:
| Just because it is in C++ it is supposed to be faster? I highly
| doubt that.
| snikolaev wrote:
| It is unlikely that it is because of C++, however, we have
| conducted extensive benchmarking (which, by the way, is fully
| open-source and can be easily reproduced if desired). You can
| find more information about this at
| https://manticoresearch.com/blog/manticore-alternative-to-
| el....
| janmo wrote:
| It is, and also it is the way c/c++ makes you write code.
|
| Languages such as Java or PHP make you lazy and you end up
| using the string variable type a lot. It is extremely
| inefficient.
| Conscat wrote:
| C/++ actually make you write relatively slow code too, by
| default. Not to the extent of Java, but still there is HUGE
| room for improvement in libC and by extension the STL. I'm
| working on a slash-and-burn approach to the problem here:
| https://github.com/cons-cat/libcat
| Minor49er wrote:
| I don't see anywhere where they claim that it's faster simply
| because it's written in C++. They do mention that they make use
| of C++ to add low level optimizations that make queries faster
| and the memory imprint smaller, but any claims about
| performance in the readme are linked to benchmarks to back up
| their claims
|
| https://github.com/manticoresoftware/manticoresearch/
|
| https://db-benchmarks.com/test-taxi/#manticore-search-vs-ela...
| alphanullmeric wrote:
| Someone tell the rust people
| Shorel wrote:
| No, just because it's in C++ does not mean it is automatically
| faster.
|
| However, with good enough algorithms and judicious coding and
| memory management, the possibility exists.
| pjmlp wrote:
| Same can be said by Java, even without direct support for
| value types.
|
| Plus it doesn't need to be Java xor C++, JNI exists for a
| reason (now Panama).
| danudey wrote:
| Also, architectural changes. They describe how ES can't
| parallelize a query unless it's spread across multiple index
| shards, which has its own tradeoffs. Their query engine can
| parallelize a query on a single index shard, which means it
| scales much more linearly on more cores without having to
| make those tradeoffs.
| pjmlp wrote:
| Which could also be done in Java.
| synergy20 wrote:
| how does it compare to another c++ elasticsearch alternative that
| was on HN a few days ago: https://github.com/typesense/typesense
| idoubtit wrote:
| In my opinion, the main difference is the history: this search
| engine has been used for 2 decades in various production sites.
|
| Mantiscore is an opensource fork of Sphinx Search, which
| released its first version in 2001. The fork started after the
| latter went from opensource to proprietary, at the end of 2017.
| The engine is stable and battle-tested. IIRC, Craiglist uses
| Sphinx.
| boyter wrote:
| I believe Craigslist has moved to manticore.
| jzawodn wrote:
| you are correct
| snikolaev wrote:
| From the Typesense's site: "Typesense is an in-memory
| datastore", "If your dataset size is 1GB, you'd need between
| 2GB - 3GB RAM to hold the whole index in memory."
|
| Manticore is different in terms of this, especially the
| Manticore columnar storage which doesn't require a significant
| portion of the data set to be stored in memory. This allows,
| for example, for a 1TB data set to be served on a standard
| server with some 32GB of RAM.
| AlexAltea wrote:
| Related: Meilisearch v1.0.0 release two days ago:
| https://news.ycombinator.com/item?id=34707727
|
| I have been following these two libraries (Manticore and
| Meilisearch) very closely. Their simplicity, portability and
| performance gains over Elasticsearch are impressive.
|
| Since two days ago, I am creating Python bindings for the core
| search engine of each of these two libraries, starting with
| https://github.com/AlexAltea/milli-py. Getting extreme
| performance, but as an embedded/self-contained package (basically
| same goals as SQLite).
| mardix wrote:
| Loving it. I'm interested in milli-py.
|
| What can be a cool feature, it's auto backup to S3, or load
| from S3.
| ollybee wrote:
| Give Xapian a go also.
| remram wrote:
| Xapian is a library but is licensed under GPL, so you can't
| build on it without making your whole app GPL.
|
| You can get around that by having the search happen in a
| separate process or something, maybe. But this is a huge
| issue for something that one might want to embed.
| kat_rebelo wrote:
| that is a slight misunderstanding of how open source
| licensing works.
|
| the GPL bleed only happens if you distribute your
| application, meaning to sell or give away binary packages
| for customers to install. if your product is a hosted api
| that you do not distribute, you do not invoke that clause.
|
| also, a lot of open source projects handle this by having
| things like the core engine licensed on a copy-left
| friendly license (GPL,AGPL). however, the language
| connectors and bindings are licensed under the slightly
| less restrictive apache license. unless you are offering a
| saas service of the product itself, it is more likely you
| are actually interacting with the connectors anyways.
| mongodb is a classic example of this model.
| AlexAltea wrote:
| It can be a problem if you intended to make an embeddable
| search engine within applications meant to be executed by
| your end-users (as is the case with _milli-py_ above).
| arein3 wrote:
| Regarding performance, hope it's not the same as Graphana's
| Loki.
|
| Grapana Loki advertises lower resource requirement, but it's
| just a disk storage system. Any query will read everyrhing from
| disk.
|
| The Elasticsearch has big RAM requirements if you create a lot
| of indexes of course. You can't have something more quick than
| indexes, and you can't have lower resource requirements without
| having fewer indexes.
| drowsspa wrote:
| What do you mean by "any query will read everything from
| disk"? Is that when you do text search or even when you
| lookup by labels Prometheus-style?
| arein3 wrote:
| Tags in loki are things like host, application, and
| environment. When searching by those tags and a time
| interval, it will read everything from disk. So any query
| that filters by ex. SessionId or a keyword from the log
| like Exception will read all the logs from disk. This can
| take ages if you have a lot of logs and a big time frame.
| Compare that with Elasticsearch which can index anyrhing,
| like SessionId/log message and return the result in an
| instant, without even reading the disk.
| m3affan wrote:
| I wonder how lasting will the support be for such libraries
| antman wrote:
| github 404 fyi
| AlexAltea wrote:
| Fixed, apologies! I had finished the PoC last night but the
| repo was still marked as private.
| davewritescode wrote:
| Anytime I see an alternative to Elastic search on HN my first
| thought is how much of a shame it is to use something other than
| Lucene for text search because of just how powerful it really is.
|
| Elasticsearch is a pain to tune and partition, and the JVM brings
| a whole set of operational issues but what's the point of better
| read/write performance when the actual search performance is
| worse?
|
| I guess this makes sense for use cases where you care more about
| speed than the quality of results.
| snikolaev wrote:
| When the search performance is worse - may be no sense.
| Regarding Manticore, we conducted relevance tests and found it
| to be on par with Elasticsearch. In fact, the objective tests
| [1] showed that Manticore can even provide better relevance
| results than Elasticsearch, when using almost default settings.
| You can view the relevant pull request in the BEIR information
| retrieval benchmark [2].
|
| [1]
| https://docs.google.com/spreadsheets/d/1_ZyYkPJ_K0st9FJBrjbZ...
|
| [2] https://github.com/beir-cellar/beir/pull/92
| idoubtit wrote:
| Do you have any sources when you claim that Lucene is the best
| search engine because it is much more "powerful" to the point
| it's a "shame" to use anything else, and because every engine
| "actual search performance is worse" than Lucene?
|
| This is a very strong claim, and without strong arguments, it's
| a ridiculous claim.
| oarsinsync wrote:
| Agreed. As much as regular expressions resulted in me having
| two problems, Lucene doesn't actually seem any better.
| dang wrote:
| Related:
|
| _Manticore Search: Elasticsearch Alternative_ -
| https://news.ycombinator.com/item?id=32261618 - July 2022 (69
| comments)
___________________________________________________________________
(page generated 2023-02-10 23:00 UTC)