[HN Gopher] Multicore OCaml: Feb 2021 with new preprint on Effec...
___________________________________________________________________
Multicore OCaml: Feb 2021 with new preprint on Effect Handlers
Author : sadiq
Score : 104 points
Date : 2021-03-11 15:09 UTC (7 hours ago)
(HTM) web link (discuss.ocaml.org)
(TXT) w3m dump (discuss.ocaml.org)
| cdaringe wrote:
| I installed one of the experimental compilers a couple months ago
| and rewrote a shell tool with effects. It worked. And I loved it.
| I can't wait. I would have kept using the experimental compiler,
| but there was compatibility issues with the rest of the
| ecosystem. The most glaring one was I couldn't use any
| preprocessing. Nonetheless, inspiring stuff.
| nwienert wrote:
| Would be interested to read the code! I'm playing with an idea
| for a shell tool and wouldn't mind trying out OCaml.
| sadiq wrote:
| As usual, happy to answer any questions.
| nullifidian wrote:
| Will there be any kind of slowdown for single-threaded code due
| to concurrent GC, or some other change needed for the multi-
| core implementation?
| sadiq wrote:
| Multicore is almost entirely backwards compatible with stock
| OCaml.
|
| One issue is C libraries that use so-called naked pointers
| but these are discouraged anyway and 4.12 ships a detector to
| find these: https://discuss.ocaml.org/t/ann-a-dynamic-
| checker-for-detect...
|
| There are also two functions on the Obj module which aren't
| supported because they're unsafe to do concurrently. These
| are clearly marked as "not for the casual user" so their use
| is relatively rare.
|
| With regards to performance, you can find a fairly detailed
| analysis in our ICFP 20 paper:
| https://arxiv.org/abs/2004.11663 In short, the impact is 3-4%
| across those benchmarks.
|
| You can grab the current multicore compiler on
| https://github.com/ocaml-multicore/ocaml-multicore and give
| it a spin to see, today.
| Blikkentrekker wrote:
| It was my understanding that _OCaml_ without a global runtime
| lock had existed for quite a while and could be optionally
| enabled, but in practice only made things slower as it
| significantly slowed down garbage collection. -- was my
| understanding wrong?
| anuragsoni wrote:
| > was my understanding wrong?
|
| I believe so. There hasn't been any ocaml release where you
| can opt in to removing the global runtime lock.
| dgllghr wrote:
| I'm very excited for this! I remember seeing a presentation a
| few years ago that mentioned that OCaml could use effect
| handlers to implement preemption in fibers. I can't seem to
| find that presentation, but I believe it mentioned being able
| to interrupt a fiber with an effect. Is that something that is
| in the plans or was that just an early idea?
| sadiq wrote:
| Was this the paper? https://www.repository.cam.ac.uk/bitstrea
| m/handle/1810/28323...
|
| Effects that 'interrupt' a fiber are asynchronous effects.
| They can be used to implement schedulers and the like. We
| don't currently support them in Multicore, there's still a
| few open research questions as to how to reconile them with
| the effect system.
| dgllghr wrote:
| Cool! Glad to hear there is still work being done there. I
| was able to find the presentation I was thinking of:
| https://kcsrk.info/slides/handlers_edinburgh.pdf
| Specifically the part about Preemptive Multithreading
| JulianMorrison wrote:
| Do you have an idea when, roughly, main line OCaml going to be
| fully multicore? (Or even a close/not close judgement.)
| sadiq wrote:
| It's pretty close. There have been PRs landing to close the
| gap with Multicore for a couple of releases now.
|
| You should see an OCaml 5.0 branch (which brings parallelism
| via Domains) in the next couple of months. The time between
| that branching and release depends a lot on how the review
| process for the Multicore patches go though.
| agumonkey wrote:
| Are there projects built on top of this branch already ?
| sadiq wrote:
| I'm not aware of large standalone projects that depend
| solely on it. Multicore is backwards compatible with
| stock OCaml though and 4.12 ships with some stub Atomics
| (https://github.com/ocaml/ocaml/blob/4.12/stdlib/atomic.m
| li) which makes writing programs that can work on both
| easier.
|
| We also maintain some projects on top of Multicore that
| are pretty interesting:
|
| * Domainslib - channels and task queue that makes using
| parallelism via Domains much easier:
| https://github.com/ocaml-multicore/domainslib
|
| * eioio - very much a work in progress but enables high
| performance direct-style IO with fibers and effects.
| Backs on to io_uring on Linux and is designed to also
| work with IOCP on Windows: https://github.com/ocaml-
| multicore/eioio
| agumonkey wrote:
| interesting ~name collision
| http://eecs.ucf.edu/~leavens/emacs/eieio/eieio-custom.el
|
| :) #ocam.el
|
| thanks for answering (and for multicore)
| eternalban wrote:
| I want to use OCaml for professional development but have
| always been held back because of hearsay around tooling and
| libraries that cast doubt on the reasonableness of the choice.
| Care to advocate for that? (No need to sell the language
| itself; I like it.)
| fezzez wrote:
| Not the original poster but I think the best answer to that
| would be from a talk DHH (the guy who made rails) gave about
| language choice.
|
| He pointed out out that when he started his company there was
| no rails, or any great support for writing a server in ruby.
| He pointed out that it's actually surprisingly simple to get
| a barebones version of whatever library you like done in your
| preferred language. The original working version of rails was
| less than 2000 lines of code. Sure, there are no bells and
| whistles, but that's how rails started, and look where it is
| right now.
|
| Compared to the ruby ecosystem in 2004, the Ocaml ecosystem
| actually pretty full of great libraries. Yeah, it's still
| tiny in comparison to the big server languages, but it's more
| than enough to build reliable systems on top of.
| Shoop wrote:
| dune is quite weird compared to many other build tools, but
| once you learn how it works it's actually quite simple. opam
| is a pain in the ass but it's definitely being improved
| steadily and I've had better experiences with it recently.
| janestreet core seems more stable now than it has in the past
| but there are occasionally still some breaking changes. They
| aren't hard to keep up with if you update with each core
| release though (only like ~1 for my project in the past year
| or so). There's also a community movement to work a lot more
| on the ocaml platform and I'm super excited for this work [1]
| to get through the pipeline! Especially promising is the work
| on replacing ocamldoc (which sucks) with odoc (which looks
| great!). While I think work on the ocaml platform will take
| some time to come to fruition, the approach they are going
| for is super empathetic and I wholeheartedly agree with their
| approach for every facet of the ocaml ecosystem -- build
| tools, doc generation, lsp, editor integration, opam
| evolution, and all the stability guarantees they're gradually
| adding. Really exciting time to invest in ocaml!
|
| [1] https://www.youtube.com/watch?v=E8T_4zqWmq8
| ithrow wrote:
| The ecosystem is fragmented between four competing "standard
| libraries" that overlap in functionality but each one has
| some feature the other doesn't have. So you can imagine the
| mess and frustration all this causes.
| eternalban wrote:
| (Thanks to all who replied but yes, mainly this.)
|
| My dated impression is that you had to choose between Jane
| Street or the alternative. And the concern was that it
| didn't seem clear how to make this decision without first
| investing significantly in the language. So this
| possibility of 'choosing the wrong horse', in my mind, made
| it a more costly proposition.
| anuragsoni wrote:
| The story isn't as dire as it sounds like. Yes there are
| multiple stdlib alternatives, but I wouldn't classify any
| of them as a "wrong" choice. They interoperate pretty
| well, and using one stdlib alternative will not block you
| from consuming libraries where the maintainer chooses
| another stdlib alternative.
|
| I tend to look at this situation as there are multiple
| really high quality options to pick from. Base [1], and
| core [2] and containers [3] are the three options I
| typically recommend people to look into (if they are
| interested in options outside of the stdlib), and all
| three are really high quality. Your decision could end up
| being as easy as deciding which platforms are important
| for you, if you don't care about windows and need to work
| with unix apis then core or containers are excellent
| options. If you do care about windows, then starting with
| Base/Containers is an excellent choice.
|
| [1] https://opensource.janestreet.com/base/
|
| [2] https://opensource.janestreet.com/core/
|
| [3] https://c-cube.github.io/ocaml-containers/
|
| Some additional notes:
|
| * Core can be seen as an extension to base as it has a
| very similar API (It re-exports many modules from base),
| and it adds some unix related utilities and additional
| data structures.
|
| * The built in Stdlib in OCaml has also been improving
| and if you are able to use a recent compiler version you
| might not feel the need for an stdlib alternative.
|
| * Containers is an excellent "default" option as it tries
| to provide a similar API as the stdlib and works on all
| major platforms supported by OCaml.
| eternalban wrote:
| Thank you, this was helpful.
| wtetzner wrote:
| Yeah, for the stuff I've been doing, just sticking with
| the built-in stdlib has been good enough.
| sleepydog wrote:
| With the caveat that I've only worked on toy projects,
| I've found the built-in stdlib to be perfectly adequate.
| I supplement with some well-known modules from dbuenzli@
| and haven't felt like I was missing any basic
| functionality.
| yawaramin wrote:
| Sure, here's some positive hearsay: OCaml tooling and
| libraries are good. Now you can use it without concern ;-)
| Scarbutt wrote:
| GP probably meant questions specific to Multicore OCaml
| xfer wrote:
| If your professional environment is windows then yes it is in
| a bad state, otherwise the tooling is pretty good.
| philzook wrote:
| dune, opam, VS Code Ocaml Platform are pretty solid trifecta
| tooling wise. dune confuses me, but what build tool doesn't?
|
| Library-wise we use bap and core as our main things. The
| documentation of both is rather hard to traverse through in
| my opinion, but I get by.
|
| Otherwise depends on what your domain is. The owl ecosystem
| looks fun https://ocaml.xyz/book/ .
___________________________________________________________________
(page generated 2021-03-11 23:02 UTC)