[HN Gopher] MillenniumDB: Property graph and RDF engine, still i...
___________________________________________________________________
MillenniumDB: Property graph and RDF engine, still in development
Author : robsalasco
Score : 57 points
Date : 2025-01-31 14:48 UTC (8 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jitl wrote:
| Is it any good?
| WhatIsDukkha wrote:
| Here is a bug with some back and forth between millenniumdb and
| qlever in starting a benchmarking attempt but I don't see
| results, though they managed to build and import.
|
| https://github.com/MillenniumDB/MillenniumDB/issues/10
|
| https://github.com/ad-freiburg/qlever
| UltraSane wrote:
| What is a domain graph?
| leetrout wrote:
| Weird title here. The repo says "Property Graph and RDF engine,
| still in development" with no mention of domain.
| dang wrote:
| We've changed the title to that of the page. (Submitted title
| was "MillenniumDB: A graph database engine using domain
| graphs")
| smarx007 wrote:
| I think if someone is just trying out RDF, it is better to start
| with Apache Jena/Fuseki or Eclipse RDF4J. Maybe
| https://github.com/oxigraph/oxigraph if you like to live
| dangerously (i.e. to use pre-1.0 DBMSs).
|
| Use of other systems involves factoring tradeoffs and
| considerations that are probably not the best for the newcomers.
| For example, qLever mentioned here is good in query performance
| and relative disk use but once the import is done, it's
| essentially a read-only DB and completely unsuitable for a
| typical OLTP scenario.
|
| Having said that, the Chilean research group that is driving the
| development of MilleniumDB is very well-regarded in the
| RDF/semantic web querying space.
| FjordWarden wrote:
| If you expect Jena to be more battle-tested because it is
| older, forget it, if the process is killed by a unexpected
| shutdown or some other reason it results in data corruption. At
| least this was my experience a few years ago.
|
| I found graph databases a beguiling idea when I first learned
| about them, and this is a welcome addition, but I've since
| temperated my excitement. They are not as flexible and
| universal a modal as is often promised. Everything is a graph,
| sure but the result of your SPARQL query not necessarily.
|
| I found classical DBMS based on sets/multisets to be much
| easier to compose from a querying point of view. A table is a
| set/multiset and a result of a query is also a set/multiset,
| SPARQL guarantees no such composability. Maybe, if you want to
| start mucking around with inference engines, but you'll either
| run into problems of undecidability.
| zozbot234 wrote:
| > SPARQL guarantees no such composability.
|
| SPARQL has a CONSTRUCT clause which gives you RDF as your
| query output. Isn't that compositional enough?
| FjordWarden wrote:
| Ok, that is true, but how do I tell my graph database that
| the result of the construct query is some other graph in my
| DB?
| smarx007 wrote:
| > how do I tell my graph database that the result of the
| construct query
|
| I am assuming you are asking how to do a CONSTRUCT query
| that will return you the contents of a given named graph?
|
| https://www.w3.org/TR/sparql11-http-rdf-update/#http-get
| is a much simpler way to get a graph. As the spec says,
| it's equivalent to the following query
| CONSTRUCT { ?s ?p ?o } WHERE { GRAPH <graph_uri> { ?s ?p
| ?o } }
| PaulHoule wrote:
| Jena lets you make little in-memory triple stores that you
| can use the way people use the list-map-scalar trinity. I've
| been working on this publication about that (RDF for
| difficult cases and when ordering counts) for years and it
| just got published last week
|
| https://www.iso.org/standard/76310.html
|
| I'll call out my collabortor Liju Fan for being the only
| person I've met who knew how to do anything interesting with
| OWL. (Well, I can do interesting things now but I owe it all
| to her.)
|
| (For the research for that paper I used rdflib under PyPi
| because CPython was not fast enough.)
|
| When I needed big persistent triple stores (that you use the
| way you might use postgres) I used to use
|
| https://en.wikipedia.org/wiki/Virtuoso_Universal_Server
|
| and had pretty good luck if I loaded a billion triples if I
| used plenty of 'stabilizers' (create a new AWS instance with
| ample RAM, use scripts to load a billion triples starting
| from an empty database, shut it down, make an AMI, start a
| new instance with the AMI, expect it to warm up for 20
| minutes or so before query performance is good)
|
| I don't regularly build systems on SPARQL today because of
| problems with updating. In particular, SQL has an idea of a
| "record" which is a row in a table, document oriented
| databases have an idea of a "record" which is a bit more
| flexible. Updating a SPARQL database is a little bit
| dangerous because there is no intrinsic idea of what a record
| is; i mean, you can define one by starting at a particular
| URI and traversing to the right across blank nodes and saying
| it is a 'record' and it works OK. But it's a discipline that
| I impose on it with my libraries, it ought to be baked into
| standards, baked into the databases, wrapped up in
| transactions, etc. For anything OLTP-ish I am still using SQL
| or document-oriented databases, but I hate the lack of
| namespaces and similar affordances that make SPARQL scalable
| in terms of "smash together a bunch of data from different
| sources" in document-oriented databases wheras SPARQL is
| missing the affordances you have in document-oriented
| databases for handling ordered collections. We badly need a
| SPARQL 2 which makes the kind of work that I talk about in
| that technical report easy.
| svilen_dobrev wrote:
| datomic (and partially xtdb /former crux) are OLTPish, and
| use only such "tuples" , essentially it's up to the user to
| define what constitutes an entity if at all ("row",
| "object", "document", whatever) - maybe some entity-id and
| everything linked to it, but maybe other less-identity-
| related stuff. Which might feel freeing to extent, but as
| you said, also expects great responsibility/discipline to
| cobble the proper properties together.
| PaulHoule wrote:
| Mathematically the boundaries of a record can be defined
| by production rules
|
| https://en.wikipedia.org/wiki/Business_rules_engine
|
| which could be written as SPARQL queries, I've used these
| to cut records out of a big graph, I haven't thought
| seriously if these could be built into a large scale
| general purpose systems.
|
| The most fun I ever had with Jena was when I used the
| rules engine for the control plane of a batch processing
| system which used stream processing primitives [1]
|
| https://jena.apache.org/documentation/inference/
|
| The Jena folks said my use was completely unsupported, I
| had looked at the source code and got to understand how
| the rules engine worked and I knew damn well there was
| nothing wrong with what I was doing.
|
| I've thought a lot about why production rules have had so
| little impact on the industry, I mean people really hate
| drools
|
| https://www.drools.org/
|
| That kind of system is particularly strong at handling
| deep asynchrony, like when a business process at a bank
| might involve some steps where you might have to wait for
| a loan office to approve a loan. It's disappointing to me
| that nobody has tried to use them (so far as I can tell)
| to deal with the asynchronous comms problems in
| Javascript though I've yet to get a clear picture in my
| mind about how to get started on that. (Funny I am
| getting an idea now so I'm putting a ticket on my
| personal Kanban board)
|
| [1] I worked later at a place that had a similar engine
| written in very awkward Scala that allegedly used Either
| and Optional for error handling but actually dropped
| errors most of the time; I knew what algebra my engine
| supported, they argued whether or not something like that
| had an algebra; my engine got the same answers every time
| because it tore down the system properly at the end,
| their engine gave different answers every time but they
| didn't seem to care
| smarx007 wrote:
| > Updating a SPARQL database is a little bit dangerous
| because there is no intrinsic idea of what a record is
|
| SPARQL has a notion of a transactional boundary just like
| SQL has. You can combine multiple SPARQL queries in one
| transaction, they will all succeed or all fail just like
| you'd expect.
| PaulHoule wrote:
| Sorta kinda.
|
| Your code has to put the right things in a transaction
| all the time for transactions for transactions to work
| right. If there is some flow of information like
| application does query -> application thinks ->
| application does update
|
| you have to wrap the whole sandwich in a transaction,
| people frequently don't do that. If I'm writing 20 of
| those for an application I want something that I know is
| bulletproof.
|
| My experience with SQL is that the average SQL developer
| doesn't really understand how to do transactions right
| but their ass gets saved (in a probabilistic sense) by
| the grouping of updates that is implicit by running an
| INSERT or an UPDATE against a table.
|
| There's also the fact that a lot of triple stores are
| seriously half baked research-quality code if that. Many
| triple stores struggle if you just try to load 100,000
| triples sequentially, for an application like my YOShInOn
| RSS reader which I expect to use every day and not have
| to patch or maintain anything for 18+ months. (Ok, a 20GB
| database that needs to be pruned crept up on me
| gradually, but that's an arangodb problem, I'd expect the
| average triple to store to have crumbled 17 months ago.)
|
| I'd love to have something that updates like a document-
| oriented database but lets you run a SPARQL query against
| the union of all the documents. Database experts though
| always seem to change the subject when it comes to having
| a graph algebra that lets you UNION 10 million graphs.
|
| (For that matter, I sure as hell couldn't pitch any kind
| any kind of "boxes-and-lines" query tool [1] etc. that
| passed JSON documents/RDF graphs over the lines between
| the operators to the VCs and private equity people who
| were buying up query engines circa 2015 because they were
| hung up on the speed of columnar query engines... Despite
| the fact that the ones that pass relational rows over the
| lines require people who really aren't qualified to do so
| create analysis jobs that look like terrible hairballs
| because of all the joins they do.)
|
| [1] Alteryx, KNIME
| smarx007 wrote:
| > you have to wrap the whole sandwich in a transaction
|
| True, SPARQL does not allow "opening" transactions such
| that you can run one query, do some logic, and run
| another query while doing commit. Which was a pain for
| me. RDF4J has a non-standard API to do that, I think they
| are trying to upstream it to SPARQL 1.2.
|
| > There's also the fact that a lot of triple stores are
| seriously half baked research-quality code if that.
|
| Also true. Although excellent researchers who wrote one
| of the best reasoners (Pellet) decided to leave academia
| and make a production grade system. They succeeded with
| Stardog but you don't want to know how much a license
| costs.
|
| > couldn't pitch any kind any kind of "boxes-and-lines"
| query tool [1] etc. that passed JSON documents/RDF graphs
|
| I really enjoy this talk from one of the creators of OWL
| [1]. There, he makes a point that OWL is unpopular not
| because it's too complex but because it's not advanced
| enough to solve real problems people care about (read:
| ready to pay money for). I think the case you described
| involves VCs having clarity on how to make money off one
| thing but not the other. I do think that the Semantic Web
| 3.0 (if we count Linked Data as a Semantic Web 2.0 aka
| Semantic Web Lite attempt) will need a better (appealing
| to business) case than the one presented in the 2001
| SciAm paper.
|
| [1]:
| https://videolectures.net/videos/eswc2011_hendler_work
| PaulHoule wrote:
| Personally I thought Stardog was trash, but if I'd had
| different requirements I might be happy with it.
|
| The trouble w/ OWL as I see it (talked about in that TR)
| is that people don't really want "first order logic", but
| they want "first order logic + arithmetic" which is a
| nightmare that Kurt Godel warned you about. (That ISO
| 20022 which that TR is related to is about the financial
| domain which is all about arithmetic)
|
| After Doug Lenat's death a lot of stuff came out that
| revealed the problems w/ Cyc, not least that even if you
| try to build something that is "knowledge based" it can't
| practically solve all the problems you want using a SMT-
| based strategy but you have to build a library of special
| purpose algorithms for everything you want to do and it
| turns out to be a godawful mess.
|
| I'm disappointed that the semweb community hasn't made a
| serious crack at usable and efficient production rules
| (dealing w/ problems like negation, controlling execution
| order, RETE execution, retraction) instead we get half-
| answers like SPIN with fixed-point execution (used an
| even more half-baked version of that to research that TR,
| gets you somewhere). Of course, production rules never
| got standardized in any domain because nobody can agree
| on _the_ way to address those four issues even though it
| usually isn 't hard to find an answer that's fine for a
| particular application.
|
| (It's a frequently problem that experts on a technology
| can get by on half-baked specific answers that would need
| a general solution if they were going to be useful for a
| general audience. One reason why parser generators are so
| bad is that if you understand parser generators enough to
| write a parser generator you aren't bothered by the
| terrible developer experience of parser generators.)
| smarx007 wrote:
| I said suitable for newcomers aka people touching RDF for the
| first time. If you want production-ready, you probably want
| Stardog, Ontotext GraphDB, or AWS Neptune - neither is cheap.
| https://github.com/the-qa-company/qEndpoint is also an
| interesting project that's used in production.
| hobofan wrote:
| As someone that has built production systems with Oxigraph (and
| a bit less with Jena), I'd recommend Oxigraph over Jena any
| day. Especially if you have you are working with a Rust-based
| tech stack.
|
| You can save so much time and headache based on less
| operational complexity and the architectural options it opens
| up. If you only reinvest part of that into building a framework
| for versioning/backups, etc. you'll have a much better overall
| package.
| iddan wrote:
| Systems like Apache Jena are not production ready for anything
| serious. It makes total sense to start something different
| spothedog1 wrote:
| Definitely do not start with Jena/Fuseki, pain in the ass to
| set up. Start with Oxigraph or rdflib in memory to play around
| with how to query/interact with the graphs
| jerven wrote:
| MilleniumDB is an interesting engine, as is Qlever mentioned in
| other comments. I think both are good candidates at making RDF
| graphs one or two orders of magnitude cheaper to host as sparql
| endpoints.
|
| Both seem to have arrived at the stage of transitioning from
| research to production code.
|
| Very exiting for those of us providing our data in RDF and
| exposing Sparql.
|
| AWs Neptune analytics is also very interesting, allowing Cypher
| on RDF graphs. Even the Oracle inbuilt RDF+Sparql seems to have
| improved greatly in 23ai.
| j-pb wrote:
| These guys write really great papers!
|
| We implemented a simplified version of their ring index for our
| data space (https://github.com/triblespace/tribles-
| rust/blob/master/src/...), and it's a really simple and cool idea
| once you wrap your head around it. Funnily enough, we build this
| even before the paper was officially published, because we found
| a preprint on one of the authors blogs. The idea itself was
| published by them before but their new paper made this a lot
| easier to understand. (burrows wheeler transforms vs. stable
| column sorting).
|
| It's really too bad that the whole linked-data space is
| completely gunked up with RDF.
|
| Ps: If anyone plans on implementing their ring index, using 0
| based offsets makes the formulas much more streamlined, their
| paper uses 1 based indexing and they have to +/-1 all over the
| place.
___________________________________________________________________
(page generated 2025-01-31 23:00 UTC)