[HN Gopher] Drasi: Microsoft's open source data processing platf...
___________________________________________________________________
Drasi: Microsoft's open source data processing platform for event-
driven systems
Author : benocodes
Score : 166 points
Date : 2024-10-20 16:07 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| gigatexal wrote:
| Oh this very much reminds me of [feldera](https://feldera.com) --
| they do incremental loads and computations using some novel
| approaches (most of which i am too dumb to follow). Really nice
| folks too.
| woozyolliew wrote:
| Or the related Materialize stuff https://materialize.com/
| hobofan wrote:
| I took a brief look into Drasi and it looks like it doesn't
| do any of the differential/timely dataflow stuff (like
| Materialize does), or any other sophisticated incremental
| view maintenance methods that are rooted in Microsoft
| Research.
| smarx007 wrote:
| https://azure.microsoft.com/en-us/blog/drasi-microsofts-newe...
| CharlieDigital wrote:
| Very interesting choice of using Cypher[0]
|
| In 2014, we built a similar type event-driven system (but
| specifically for document distribution (a document can be
| distributed to a target set of entities; if a new entity is
| added, we need to resolve which distributions match)) and also
| ended up using Cypher via Neo4j (because of the complex
| taxonomical structure of how we mapped entities).
|
| It is a super underrated query language and while most of the
| queries could also be translated to relational SQL, Cypher's
| linear construction using WITH clauses is far, far easier to
| reason about, IMO.
|
| EDIT: feel like the devs went overboard with the mix of
| languages. Shoehorned in C# Blazor? Using JS and Jest for e2e
| testing?
|
| [0] https://drasi.io/reference/query-language/
| JanSt wrote:
| I too have great memories of cypher. Such an elegant way to
| write queries.
| CharlieDigital wrote:
| If you haven't been following it, I recently found out that
| it is now supported in a limited capacity by Google
| Spanner[0]. The openCypher initiative started a few years
| back and it looks like it's evolved into the (unfortunate
| moniker) GQL[1].
|
| So it may be the case that we'll see more Cypher out in the
| wild.
|
| [0] https://cloud.google.com/spanner/docs/graph/opencypher-
| refer...
|
| [1] https://neo4j.com/blog/cypher-gql-world/
| leeoniya wrote:
| > while most of the queries could also be translated to
| relational SQL, Cypher's linear construction using WITH clauses
| is far, far easier to reason about, IMO.
|
| https://prql-lang.org/
| CharlieDigital wrote:
| Didn't look too deeply, but one of the keys with Cypher (at
| least in the context of graph databases) is that it has a
| nice way of representing `JOIN` operations as graph
| traversals. MATCH
| (p:Person)-[r]-(c:Company) RETURN p.Name, c.Name
|
| Where `r` can represent any relationship (AKA `JOIN`) between
| the two collections `Person` and `Company` such as
| `WORKS_AT`, `EMPLOYED_BY`, `CONTRACTOR_FOR`, etc.
|
| So I'd say that linear queries are _one_ of the things I like
| about Cypher, but the clean abstraction of complex `JOIN`
| operations is another huge one.
| robertlagrant wrote:
| We made a health backend partly using Cypher and the only thing
| I found was the simple queries looked amazing, but as soon as
| you need to join non-linearly it started looking a lot like SQL
| again. And when you're using an ORM it stops mattering. And
| when you need migrations it gets painful!
| CharlieDigital wrote:
| > but as soon as you need to join non-linearly
|
| At least in our use case, even with some very gnarly 20+ line
| Cypher queries, it never got to the point where it felt like
| SQL and certainly, those same queries would be even gnarlier
| as nested sub-selects, CTEs, or recursive selects, IMO.
|
| Perhaps a characteristic of our model (a taxonomy of Region,
| Country, Sponsor, Program, Trial, Site, Staff for global
| clinical trials and documents required by
| Region/Country/Program/Trial).
| fatliverfreddy wrote:
| I wish I could use Cypher for everything
| f4c39012 wrote:
| Purple!
| computronus wrote:
| Green!
| JaimeThompson wrote:
| Just my luck. I get stuck with a race that speaks only in
| macros.
| imvetri wrote:
| What does it process it from and what does it process it to?
|
| Is it programmable or you have a concrete concept theorised?
|
| What is it useful for? How it helps business in saving cost or
| increasing profit? Is it a hobby project?
| otterley wrote:
| Looks very Azure-centric. Both installation guides
| (https://drasi.io/how-to-guides/install-sample-applications/b...
| and https://drasi.io/how-to-guides/install-sample-
| applications/c...) require Azure to work.
|
| And then there's this:
|
| > Installing Drasi in an EKS cluster can be significantly more
| complex than a standard installation on other platforms. Instead
| of downloading a CLI binary using the provided installation
| scripts, this approach requires modifying the source code of the
| Drasi CLI and building a local version of the CLI.
|
| Is this an actual requirement or just the current easy path?
| dtquad wrote:
| That is usual for new Microsoft open source projects. It takes
| 1-2 months for the Azure dependencies to go away.
| pjmlp wrote:
| Azure is the new Windows, as timesharing OS, thus yeah that is
| to be expected.
| jameslevy wrote:
| Does it require Azure to work? Or could the Azure steps be
| relatively easily be swapped out for AWS/GCP/etc?
| stackskipton wrote:
| Azure SRE here, it doesn't appear to have any Azure
| dependencies. CLI rebuild seems to be that "drasi init" assumes
| Azure Kubernetes Service built in StorageClasses for Kubernetes
| PVC for Redis and Mongo and thus fails when running against
| EKS. I assume same thing would be required on GKE. Yes, it
| should be more modular but MVP.
|
| As for other stuff, it's using Gremlin Query Language or
| Postgres which are both open. In fact, it's going out of way
| it's not to use Azure authenication as loading connection
| string as Kubernetes secret is 100% AGAINST Azure Kubernetes
| Best Practice. Best Practice would be Workload Identity.
| sitkack wrote:
| But at what cost?
| stefanos82 wrote:
| Drasi...React...well played Microsoft, well played :D
|
| Assuming they choose this name from the Greek drase which means
| action, React of course is the exact opposite to action, thus the
| React-ion; an action expects a reaction, somewhere somehow!
| benbristow wrote:
| Not like Microsoft to name things well...
| j-a-a-p wrote:
| VMS++ = Windows NT?
___________________________________________________________________
(page generated 2024-10-20 23:00 UTC)