[HN Gopher] The Unison language - a new approach to Distributed ...
___________________________________________________________________
The Unison language - a new approach to Distributed programming
Author : guytv
Score : 167 points
Date : 2022-11-17 12:37 UTC (10 hours ago)
(HTM) web link (www.unison-lang.org)
(TXT) w3m dump (www.unison-lang.org)
| tr33house wrote:
| how is this different from apache beam? also, any orgs using this
| at scale?
| adamgordonbell wrote:
| It's a programming language where the file representation isn't
| text of the code but ASTs and you can call things distributedly
| by a hash of that AST.
|
| So I guess its different from apache beam in all of the ways.
| muricula wrote:
| How does the runtime manage different levels of trust between
| clients? How does the language grapple with code injection
| vulnerabilities? If I want to be able to receive data from an
| untrusted entity, but not code, how do I make sure they're not
| submitting code to my server to unpack and execute?
| zawodnaya wrote:
| You can simply expose an HTTP endpoint that receives the data.
| You wouldn't want to expose the internode protocol endpoint to
| the internet.
|
| That said, it wouldn't be as bad as it sounds. Unison is a
| purely functional language, so if you don't explicitly provide
| the ability to e.g. do arbitrary I/O, then other nodes will not
| be able to send you code that does I/O. It will not type-check.
| halfmatthalfcat wrote:
| I had to dig for the distributed part but it's outlined here:
| https://www.unison-lang.org/articles/distributed-datasets/
| ofrzeta wrote:
| But the only runtime available for this sort of application
| seems to be the "unison cloud" that is a hosted service.
| alexashka wrote:
| This strikes me as a compiler and IDE feature, rather than a
| reason to have a separate programming language.
|
| What does having a separate language give us as opposed to taking
| say, Kotlin, and having a compiler that stores the AST/whatever
| in a database and does all the interesting goodies?
|
| Or is that the end goal, but we're using a basic language to test
| it out and work out all the quirks before writing a compiler for
| existing languages?
| politician wrote:
| - Function definitions stored in a content-addressed database.
|
| - Dependency management handled the same way the Nix handles it.
|
| - Some kind of object storage system that uses content-
| addressable structures as the schema.
|
| - Hyperlinked codebase.
|
| - Human-readable function names as (essentially) git tags.
|
| These ideas are all pretty nice. A dedicated IDE for this
| language would be a lot of fun to work with. The debugging story
| likewise seems like it will be pretty solid. I'm not sold on the
| zero config storage layer: basic object retrieval is different
| than schema prepared for query performance. I'd like to learn
| more about the concurrency and synchronization story.
| klabb3 wrote:
| I think that content addressability works really well for code,
| at the bottom layer. But in order to be practical, you
| typically need:
|
| Naming and resolvers, in order to be human friendly. This isn't
| easy to get right, but we have a lot of prior art in dependency
| management systems.
|
| Persistence layers with GC, like cache and db. You're gonna
| want to fetch and prefetch in ways that are quite advanced. You
| don't want to be be blocked in a critical section by network
| fetching the leftpad function.
| 6gvONxR4sf7o wrote:
| I'm excited for ideas like this to become mainstream. Today's
| approaches to heterogeneous and distributed computing are how I
| imagine single core computing was 40 years ago. You have to
| manually manage practically everything. Instead, let the compiler
| or interpreter or whatever figure out where to actually run it
| (CPU vs ALU vs GPU vs remote machine #42), what to keep in what
| part of cache (L1 vs L2 vs RAM vs disk vs S3), etc.
| junon wrote:
| We have, though. Spark, for example, does this in just about
| every language. It's been around for ages, is liberally
| licensed, and deployed at scale in thousands of enterprises.
| zawodnaya wrote:
| Well, Spark does allow you to accomplish distributed
| workloads for certain forms of computation. But it's limited
| to those forms of computations (streaming, map-reduce). It
| also has a large operational footprint. It's also lamentable
| that distributed code that uses Spark looks nothing like its
| non-distributed counterpart.
|
| Something very much like Spark map-reduce can be implemented
| in ~100 lines of Unison code:
|
| https://www.unison-lang.org/articles/distributed-datasets/
|
| Some videos on Unison's capabilities over and above Spark:
|
| Distributed programming overview:
| https://www.youtube.com/watch?v=ZhoxQGzFhV8
|
| Collaborative data structures (CRDTs):
| https://www.youtube.com/watch?v=xc4V2WhGMy4
|
| Distributed data types:
| https://www.youtube.com/watch?v=rOO2gtkoZ3M
|
| Distributed global optimization with genetic algorithms:
| https://www.youtube.com/watch?v=qNShVqSbQJM
| __MatrixMan__ wrote:
| Spark still references functions by name though, right? So if
| a peer says "call this function" I have to trust both that
| peer AND whatever system resolves the name to some cpu
| instructions.
|
| Plus you're significantly limited in how much you can memoize
| if you can have different versions of the "same" function.
| cmollis wrote:
| spark is great for distributed computation.. also has about a
| million config switches and is generally kind of 'bulky'. EMR
| makes management a lot easier, but you still have to fiddle
| with num executors, memory, etc. But it has been 'through the
| wars' and is generally pretty solid on some pretty large data
| sets. Once you get it conf'd it's pretty good. The best part
| is just writing the scala code to run the job.. admittedly,
| it would be great to use something a bit lighter for certain
| workloads.
| dwohnitmok wrote:
| How do Unison abilities handle non-commutative abilities (i.e.
| abilities where the order in which one applies handlers matters).
| Does it just assume that abilities are commutative? Or rely on
| the programmer to make sure that handlers are applied in an order
| that makes sense?
| zawodnaya wrote:
| The latter.
| gauddasa wrote:
| No description on the home page, the documentation, Github page
| ... mentions that this is a Haskell-like language inspired by
| Haskell. No credit to Haskell while the code base is 99.6%
| Haskell!
| brabel wrote:
| I think they are making a mistake that's common in this sort of
| project: trying too many new things at once!
|
| They already have a very innovative way of managing source code,
| with a database of definitions that keeps the hash of the syntax
| tree instead of actual source. That's a very neat idea that
| solves many problems (read their docs to understand why).
|
| But instead of developing that well enough so that it works with
| source control tools, IDEs, can be deployed easily and painlessly
| on existing infrastructure... no, they decided to ALSO solve
| distributed computing, a really, really complex space with a
| pretty crowded space of solutions... and seem to be focusing on
| that now instead of the "original" ideas. Looks like a huge issue
| with scope creep to me... unless they are kind of pivoting to
| distributed computing now only because the original ideas were
| not attractive enough for people to embrace it, but I have not
| heard of anything like that, everyone seems to be pretty vibed by
| those things.
| dack wrote:
| To be fair, for a while they were ALSO working on their own
| graphical source editor that allowed for type-correct
| transformations and assisted refactorings. They put that on the
| back burner specifically because they are trying to focus on
| fewer things :)
|
| I think the distributed computing problem is pretty related
| once you have "content-addressable" source code. Agreed that
| it's a lot of work but I hope it pans out!
| renox wrote:
| I disagree, there's no real relationship between 'content
| addressable' source code and distributed computing.
|
| Also I don't think that you need to create a new language to
| have 'content addressable' source code distribution..
|
| Creating yet another language ensure that this will get
| nowhere, too bad.
| zawodnaya wrote:
| Well, there is a relationship. The relationship is
| specifically that Unison nodes can communicate code with
| one another unambiguously by exchanging hashes.
| pharmakom wrote:
| Would this not work just as well with a lisp or even JS?
| stewoconnor wrote:
| (I'm the a Unison employee that works mostly on
| distributed computing)
|
| There are a few things about unison that make this
| easier:
|
| * content addressed code * unison can serialize any
| closure
|
| I can ask for a serialized version of any closure, and
| get back something that is portable to another runtime.
| So in a function, I can create a lambda that closes over
| some local variables in my function, ask the runtime for
| the code for this closure and send it to a remote node
| for execution. It won't be a serialized version of my
| entire program and all of its dependencies, it will be a
| hash of a tree, which is a tree of other hashes.
|
| The remote node can inspect the tree and ask for or
| gossip to find out the definitions for whatever hashes in
| that tree it doesn't already know about. It can inspect
| the node for any forbidden hashes (for example, you can'd
| do arbitrary IO), then the remote node can evaluate the
| closure, and return a result.
|
| In other langauges, perhaps you can dynmically ship code
| around to be dynamically exeecuted, but you aren't going
| to also get "and of course ship all the transitive
| dependencies of this closure as needed" as easily as we
| are able to.
| zubairq wrote:
| Yes it does work with JavaScript. At yazz that is exactly
| how we store is code, by the iPfs hash
| Quekid5 wrote:
| The key idea is that Unison code is in a sort of "normal
| form", such that e.g. what names you give things doesn't
| matter at all.
| klabb3 wrote:
| How about wasm? Interop de lux
| zawodnaya wrote:
| An issue with exchanging WASM is you'll have to do either
| dynamic or static linking of dependencies. Unison works
| around this by having all definitions share a global
| address space (and the address of any given code in that
| space is the hash of its syntax tree), so there is no
| linking step. Or rather, the linking is done trivially by
| the hashing.
| zawodnaya wrote:
| Well, it could be done in JS or Lisp. You'd have to
| replace all the references to dependencies in every
| function with a hash of the implementation of the
| referenced function and use some kind of global lookup
| table (which would need to be a distributed hash table or
| something). But this would be slow, so you'd need a
| compiler or some kind of processor to inline stuff and do
| CPS transforms, and by that time you've basically
| implemented a janky version of Unison.
| nerdponx wrote:
| My impression is that the original use case was distributed
| computing, and the content-addressable stuff took on a life
| of its own after that.
| klabb3 wrote:
| > Creating yet another language ensure that this will get
| nowhere, too bad.
|
| What has happened before, consistently, is that research or
| proof of concept- style languages pave the way for bigger
| players to take the ideas and incorporate them into
| existing or future mainstream languages.
| jupp0r wrote:
| The two areas of managing source code and distributed computing
| are not as disjunct as you make them in the context of Unison.
| They follow from the underlying principle of addressing
| functions not by their name but by a hash of their normalized
| syntax tree (ie their code).
|
| There are a bunch of cool implications for distributed
| computing, namely that you can easily distribute fine grained
| parts of your application across servers and that you can cache
| the result of expensive calculations.
| msla wrote:
| > They follow from the underlying principle of addressing
| functions not by their name but by a hash of their normalized
| syntax tree (ie their code).
|
| The first time the language or compiler changes such that the
| same code generates a different syntax tree they'd have to do
| something pretty fancy to avoid rebuilding the world. (That,
| plus all the usual caveats about what happens when old hash
| algorithms meet malicious actors from the future.)
| stewoconnor wrote:
| (Unison Developer here), yes we've done two such migrations
| this year. It has typically meant that we have an "are you
| ready to upgrade" message when you start up and the
| migration took less than a minute on my pretty large
| codebase. It's not a big deal
| msla wrote:
| It's a solved problem technically, certainly, but
| assuming all of the relevant source code will always be
| available ignores some social and legal issues.
| jupp0r wrote:
| It's a common approach in language design to require this
| though. Rust has made very similar design choices due to
| its (current) lack of stable ABIs. IIRC some hash of the
| compiler version is included in built libraries and
| prevents accidental linking. You need to rebuild the
| world for every toolchain upgrade.
| stewoconnor wrote:
| The relevant sourcecode is not available, we don't store
| any source code. All the relevant dependences MUST be
| availble in AST form though.
|
| I don't know what the social and legal issues might
| possibly be, though I might be missing somehting, what do
| you have in mind there?
| AnthonBerg wrote:
| If we want to set sail on the ocean, we do not take a basket
| and start work to stop it from leaking. We build a boat.
| xwowsersx wrote:
| I thought that was sort of Paul's goal, to explore a bunch of
| new ideas and paradigms. I doubt he's under any illusions that
| this language, qua this language, is going to see wide adoption
| or need to have all of its features really ironed out with a
| fixed/stable API. But I could be wrong!
| jrumbut wrote:
| Then that flips it back to being very cool that he is putting
| his energy into carrying a few batons far enough for others
| to pick them up.
| xwowsersx wrote:
| Agreed, I think it's great to have really cutting-edge,
| highly experimental sort of research languages that explore
| new paradigms. Doesn't mean you have to like the language,
| but I think it's good that some people are doing this kind
| of work as a going concern and not just some repo that was
| pushed up once somewhere and languishes. He's got some
| interesting ideas in there related to effects system, etc.
| xwowsersx wrote:
| > carrying a few batons far enough for others to pick them
| up.
|
| I love that metaphor btw. Gonna use that.
| IshKebab wrote:
| > I doubt he's under any illusions that this language, qua
| this language, is going to see wide adoption or need to have
| all of its features really ironed out with a fixed/stable
| API. But I could be wrong!
|
| They have VC funding and employees. I presume he's told
| investors that it will see at least fairly wide adoption!
| xwowsersx wrote:
| Ha! Did not know that. Not necessarily dispositive though.
| Also, I believe they are a public benefit corporation so
| that would also cut in favor of my point...I think
| dboreham wrote:
| An answer to the question "what useful thing has been build with
| Haskell"?
| flylikeabanana wrote:
| Here's a couple more that I've found useful over the years:
|
| - pandoc: convert (almost) any document format to (almost) any
| other document format
|
| - The Elm Compiler: Possibly the most widely used statically
| typed pure frontend language
|
| - XMonad: A tiling window manager for linux that I enjoyed
| using even without knowing much Haskell
|
| - Purescript: the other widely-used (for an FP language)
| statically typed pure compile-to-javascript language
| revskill wrote:
| "Each Unison definition is identified by a hash of its syntax
| tree." [0]
|
| I remember many years ago, i had the same idea on how to
| correctly version a dependency.
|
| [0]: https://www.unison-lang.org/learn/the-big-idea/
| thefaux wrote:
| Does Unison have stack traces? Using the hash of the ast as the
| only identifier seems like it would lose some useful runtime
| debugging information.
| stewoconnor wrote:
| Yes, we have stack traces. You might see some hashes in the
| middle of the stack traces, but whenever we do have a name,
| we'll show you the name. So if you are calling into an
| anonymous clousure, but functions in your call stack will
| typically have names available
| iamevn wrote:
| While hashes are the primary identifier, things also
| generally have names associated with them. (Unless you
| deliberately remove the names that is.)
| 1letterunixname wrote:
| Is this the same language from the how to create a programming
| language book?
| mgax wrote:
| I LOVE the idea but the language itself is so ugly I don't want
| to learn it. It looks really ugly. Sorry.
| zawodnaya wrote:
| What's an example of a pretty language?
| IshKebab wrote:
| I really hate the language but I have to say Python does at
| least look quite nice.
| jjtheblunt wrote:
| Scheme! :)
| zawodnaya wrote:
| Good news: the surface syntax of Unison is totally
| arbitrary and can in practice be swapped out. There's
| currently only one (Haskell-like) syntax, but in future I
| imagine there will be others. So you can imagine on
| https://share.unison-lang.org a little dropdown that lets
| you select what syntax you want to see the code in.
| christophilus wrote:
| Perl4
| optymizer wrote:
| I don't think this language is ugly and it is indeed hard to
| say which language is objectively beautiful, but I personally
| have always disliked syntax that uses the ' operator for some
| reason. It's small - it looks like a piece of dirt on the
| screen, which makes it hard to read. I imagine the same
| arguments could apply to the '.' or ',' operators, though '
| tends to appear around whitespace, while ./, around
| characters, so there's at least some contextual information
| around them.
| vitiral wrote:
| It's ML syntax. I'm also not used to it, but many who are
| consider it beautiful.
| stewoconnor wrote:
| Its certainly a turn-off for some. One thing that we have as
| a potential future effort would be to enable an alternate
| surface syntax. Since we store an AST insteead of source
| code, we could create a parser/pretty printer for another
| surface syntax. Then a user of the language who would prefer
| something that looked like python could switch our website's
| renderer to output the python like syntax!
| cochne wrote:
| I really like the idea of Unison, but unfortunately when I went
| to try it out, it was much slower than even Python. I hope they
| can make it performant!
| tomcam wrote:
| On a tangential note--do you like this home page as much as I do?
| It really draws me in as a programmer. I like all the code
| examples up front, and their mission statement ("A new approach
| to distributed programming/No more writing encoders and decoders
| at every network boundary") seems to be quite clear. To me one of
| the best landing pages I've run across.
| FpUser wrote:
| ' - is used to denote a delayed computation. This is hard to see
| / easy to miss. What it this trend to abbreviate everything down
| to a shortest. What is wrong with _delayed_ keyword for example
| Serow225 wrote:
| Does anyone have a comparison with Darklang/ecosystem and it's
| goals? Some of the things mentioned sound similar, at a high
| level.
| ingomaro wrote:
| >Other tools try to recover structure from text; Unison stores
| code in a database
|
| Just like VisualAge in the last century. Sign me up, that was
| great stuff.
| zawodnaya wrote:
| Smalltalk had a lot of good ideas!
| kybernetikos wrote:
| I loved it, except that it seemed to periodically corrupt the
| source code database which was an absolute nightmare.
| samsquire wrote:
| This project hashing of sourcecode (or AST) in the interpreter is
| a really powerful idea
|
| I plan to use it for a slightly different purpose. I want to
| implement an interpreter that is multithreaded similar to Java or
| Erlang that can send objects between shared memory without
| marshalling or copying.
|
| I had a talk with someone on HN
| https://news.ycombinator.com/item?id=32907523 about python's
| Global Interpreter Lock and we talked about how objects are
| marshalled between subinterpreters due to object identity. The
| identity of an object is defined at creation time as the hash of
| that object.
|
| If the hash of the object was the sourcecode, two interpreters
| could load the same Object hierarchy and send data by hash
| reference.
|
| I wrote a multithreaded interpreter that uses message passing to
| send integers and program counters to jump to code in other
| threads
|
| This is at https://GitHub.com/samsquire/multiversion-concurrency-
| contro...
| nuphonenuacc wrote:
| Iirc pony langs messages could work like that.
| toolslive wrote:
| Nil nove sub sole ? It reeks of COOLs (Concurrent Object Oriented
| Languages). Here's an example of the mid 90s:
| https://distrinet.cs.kuleuven.be/projects/CORRELATE/
| klabb3 wrote:
| Object oriented? Unison looks pretty functional to me. And I
| don't see the content addressability, which is arguably the
| most important part.
|
| Anyway, you'll very rarely see completely novel ideas in this
| space. Good combinations, compromises and applications matters
| a lot. Look at your favorite applied Merkle tree tool.
| nerdponx wrote:
| Perhaps not, but plenty of good ideas kick around for years or
| decades before someone finally figures out how to put them
| together right.
| jrumbut wrote:
| That is the perfect programming language landing page.
|
| The Hello World example introduces one of their concepts that you
| might not see every day, then they show a little algorithm, then
| a practical "stuff you need to get work done" example.
|
| I got an immediate sense that the language has some familiar "ML
| family" type features (like F# or Scala) but also some
| distinctive aspects.
| tomcam wrote:
| Oops I posted something similar to this. Agree completely.
| anthropodie wrote:
| Discussion from
|
| - a year ago https://news.ycombinator.com/item?id=27652677
|
| - 8 years ago https://news.ycombinator.com/item?id=9512955
| dang wrote:
| Thanks! Macroexpanded:
|
| _Unison Programming Language_ -
| https://news.ycombinator.com/item?id=27652677 - June 2021 (131
| comments)
|
| _Unison: A Content-Addressable Programming Language_ -
| https://news.ycombinator.com/item?id=22156370 - Jan 2020 (12
| comments)
|
| _The Unison language_ -
| https://news.ycombinator.com/item?id=22009912 - Jan 2020 (141
| comments)
|
| _Unison - A statically-typed purely functional language_ -
| https://news.ycombinator.com/item?id=20807997 - Aug 2019 (25
| comments)
|
| _Unison Language March Update_ -
| https://news.ycombinator.com/item?id=19528189 - March 2019 (1
| comment)
| woojoo666 wrote:
| And the one from 8 years ago:
|
| _Unison: a next-generation programming platform_ -
| https://news.ycombinator.com/item?id=9512955 - May 2015 (128
| comments)
___________________________________________________________________
(page generated 2022-11-17 23:00 UTC)