[HN Gopher] A better build system for OCaml
___________________________________________________________________
A better build system for OCaml
Author : gaws
Score : 339 points
Date : 2025-01-30 20:14 UTC (1 days ago)
(HTM) web link (blog.janestreet.com)
(TXT) w3m dump (blog.janestreet.com)
| wiseowise wrote:
| Dune is the single best name of a build system out there, nothing
| even comes close to.
| 708733454927516 wrote:
| Indeed.
|
| "He who controls the [build system] controls the universe."
| chii wrote:
| Wouldn't "spice" fit that monika better than dune?
| davesnx wrote:
| that's what' melange is lol
| Kototama wrote:
| Because software is built on shifting sands?
| firesteelrain wrote:
| Man I must be under a rock. Never heard of Dune!
|
| Cmake mostly
| tombert wrote:
| I have to respect Jane Street for proving that HM-typed
| functional programming can be production-ready and extremely fast
| when given enough love.
|
| I still occasionally hear things about how the more academic-
| styled functional languages can't work in production, but Ocaml
| shows that it absolutely can work, even with high performance
| requirements.
| kyawzazaw wrote:
| > HM-typed functional programming
|
| Mercury also uses Haskell for their backend
| https://mercury.com/
| tombert wrote:
| Also very cool!
|
| I was fortunate enough to work at Jet.com before Walmart
| completely destroyed it, which was an F# shop. I really liked
| it, and I never really felt "limited" by it.
|
| The stuff I was working on didn't have nearly the same
| requirements as high-frequency trading like Jane Street does
| though. I never did any super low-latency stuff with F#, so
| it's tough for me to say how well it fair with that kind of
| environment.
| jlarocco wrote:
| Too bad they didn't use their namesake:
|
| https://mercurylang.org/
|
| Would honestly be a lot more interesting than Haskell.
| brians wrote:
| Thank you for mentioning this, because--not having followed
| the link--I thought someone was using Mercury for a
| backend. And... I wasn't sure I was prepared to follow that
| link.
| jeofken wrote:
| Anyone coding Mercury out there?
| trenchgun wrote:
| ODASE uses it: https://www.mercurylang.org/documentation/
| papers/mc_swese_20...
|
| https://www.odase.io/
| jinwoo68 wrote:
| Standard Chartered also uses their variation of Haskell.
| tombert wrote:
| Yeah, I actually interviewed for a Haskell position there
| in 2023. They didn't hire me, but it was kind of fun to do
| technical questions in Haskell for an interview instead of
| the boring JavaScript or Python that I usually use.
| eru wrote:
| They also use regular Haskell these days.
| ofrzeta wrote:
| Why does Mercury not use ... Mercury?
| https://mercurylang.org/
| ablob wrote:
| There's also pandoc written in Haskell. Facebook used it for
| it's spam-protection too, at some point (it might still do
| that, but I don't know).
| philjohn wrote:
| Sadly I don't think it's used any more - it was migrated away
| from. I worked in integrity there and quite enjoyed whenever
| I had to make some changes to the Haskell rules :)
| tome wrote:
| They also have Glean:
| https://github.com/facebookincubator/Glean
| ibejoeb wrote:
| Certainly. There's also a good amount of F# in production. It's
| a fantastic option for building high quality programs with
| broad interop with existing systems.
| perryprog wrote:
| NoRedInk[0] also uses some Haskell in their backend, as well as
| Elm for most of their frontend. They've also worked some with
| Roc according to a blog post from a few years back; not sure if
| using it.
|
| [0] https://blog.noredink.com
| fulafel wrote:
| Eh, this reads like damning with faint praise to me. Lots of
| robust and fast real world systems are in FP languages. Ocaml
| is estabilished and mature, far from "is it production ready?"
| category.
| tombert wrote:
| Not really meant to be "damning", I'm a pretty big functional
| programming advocate (I've spoken at the Lambda Days
| conference four times and the Clojure Conj once).
|
| It feels like the trend right now is to bolt on one or two
| "functional libraries" into your "normal" language and
| pretend that that's the same as writing Haskell or Ocaml.
| People have actually expressed such sentiments to me because
| Java has the optional type and a "map" function for the
| Streams API. When I suggest writing something in a functional
| language, the response is always "it's too hard" or "we won't
| be able to hire for that", as if engineers are somehow unable
| to learn new things.
| emoII wrote:
| What do you mean? Afaik the type system does not affect run
| time speed, it's just static analysis
| imjonse wrote:
| dynamic typing incurs runtime overhead
| bjoli wrote:
| That is obviously not what they meant since ocaml is not
| dynamically typed.
| dkersten wrote:
| HM is type inference for static typing, though, so wouldn't
| have runtime overhead?
| wk_end wrote:
| If anything the type system improves the run time speed,
| because the static analysis enables better code generation.
|
| But I think what OP meant was more about the "functional
| programming" side of things than the "HM-typed" side of
| things. Naively, anyway, you might think that "the FP-style"
| of avoiding mutation and preferring recursion would require
| lots of garbage and high-latency garbage collection, copying,
| function call overhead...of course, that's not the whole
| story, but having Jane Street to point to as a crushing
| counter-example is nice.
| IshKebab wrote:
| I dunno if anyone really doubted HM, functional programming
| could work. I've written a bit of OCaml and apart from the bad
| syntax (code very easily becomes a word soup) it's not
| especially different to other programming languages, just more
| convenient sometimes.
|
| I think it's _purity_ that actually is a big difference, but
| OCaml isn 't pure.
| zokier wrote:
| kinda surprising that it still makes sense to develop your own
| new build system, and then do the migration to it, instead of
| adopting an off-the-shelf one like buck2 (or bazel/etc...). Not
| saying that these fancy build tools are easy to adopt, but
| compared to building your own? is there something remarkable that
| makes dune much different?
| wk_end wrote:
| Without getting into any specifics of it - I'm sure there's
| people with much more experience with these tools who can
| comment - I'll point out that neither buck nor bzl existed when
| JS decided to start building their own tool in 2012. Bazel's
| first release was in 2015, Buck's was in 2013.
|
| JS does have a bit of a NIH culture, but I'm not sure if that
| was really at play here. There just...weren't very many good
| build tools available at the time, particularly for a company
| using an unorthodox tech stack.
| zokier wrote:
| > I'll point out that neither buck nor bzl existed when JS
| decided to start building their own tool in 2012. Bazel's
| first release was in 2015, Buck's was in 2013.
|
| But Dune started (according to this blog post) in 2016 and JS
| started seriously improving and adopting it last year. So to
| me Jenga sounds like a reasonable step in 2012, but pouring
| significant effort into migrating from Jenga to Dune (and
| improving Dune) in 2024 sounds more weird
| yawaramin wrote:
| Jenga and dune are the same thing, it was just renamed.
| zokier wrote:
| The blog post clearly describes them as two different
| systems, and how Jane Street migrated from one to
| another.
| wk_end wrote:
| Yes and no. This is all spelled out in the post, but it's
| a little thorny.
|
| Dune is a rename of Jbuilder (2016). Jbuilder uses Jenga
| (2012) configuration files.
|
| > By 2016 we had had enough of this, and decided to make
| a simple cross-platform tool, called Jbuilder, that would
| allow external users to build our code without having to
| adopt Jenga in full, and would release us from the
| obligation of rewriting our builds in OCamlbuild [...]
| Jbuilder understood the jbuild files that Jenga used for
| build configuration.
|
| So in 2012 it made sense for them to build Jenga, because
| there weren't any good alternatives - Bzl etc. didn't
| exist, so they couldn't have solved their problems.
|
| And in 2016 they had open-source code they wanted others
| to be able build; those people didn't want to use Jenga,
| and JS didn't want to rewrite their builds so that they
| could use something else. Thus, Jbuilder was a shim so
| that JS could still use their Jenga builds and others
| could build JS' code without using Jenga. Bzl etc., even
| though they existed, wouldn't have solved these problems
| either.
| yawaramin wrote:
| My bad, dune is a rename of jbuilder indeed. Not Jenga.
| But the other reply provides more context that's
| important.
| mhh__ wrote:
| I don't know ocaml particularly well but my understanding is
| that only with buck2 (of those mentioned) do you have a build
| system with the semantics to actually compile ocaml properly.
|
| In particular I was under the impression one needed to be able
| to run ocamldep before hand (or compile twice) - buck2 can do
| this, bazel needs hacks iirc.
| rwmj wrote:
| I loath autotools with a passion, and it's awkward to set it
| all up, but it handles OCaml code just fine. We use it for
| multi-language projects (including OCaml) like nbdkit &
| libguestfs.
| codr7 wrote:
| I spent several years in Haskell land a while back, my first
| functional experience after Lisp.
|
| Ocaml never clicked for me, I have a rare form of semicolon
| allergy and Haskell just looked a lot nicer to me.
|
| But then I recently tried Reason and enjoyed it A LOT, so
| everything Ocaml is suddenly interesting.
| whimsicalism wrote:
| i wish the reason syntax was the main one
| cwmoore wrote:
| "If your intention is to compile or interpret files as scripts,
| double semicolons can and should be avoided when writing in
| OCaml. Leaving them does not raise errors, but they are
| useless. The compiler tolerates them to allow copy-paste from
| UTop to a file without having to remove them."
| https://ocaml.org/docs/toplevel-introduction#loading-librari...
|
| https://reasonml.github.io/ looks cool, OCaml with javascript.
| Taikonerd wrote:
| > _https://reasonml.github.io/ looks cool, OCaml with
| javascript._
|
| The Javascript-oriented part of ReasonML got forked to be its
| own language: Rescript. https://rescript-lang.org/
| disconcision wrote:
| you can, for now at least, still use reasonML with
| js_of_ocaml to compile to javascript if you like reason
| syntax but don't want to a more complicated relationship
| with the ocaml ecosystem (we use this for
| https://hazel.org)
| cmrdporcupine wrote:
| It's funny because the thing that always turned me off from
| Haskell was that it feels like a stream of whitespace and
| identifies without good visual cues as to structure.
|
| I'm sure once you get the zen of it, it's fine. Like Lisp, I
| guess, you learn to think in its structure. But looking at a
| screenful of Haskell to me is intimidating.
|
| Of the bunch I found SML/NJ to be the most readable.
| eru wrote:
| The whitespace is the visual cue.
|
| But in any case, you can use curly braces and semicolons in
| Haskell just fine. You can also write your Haskell like Lisp,
| and add lots of parens everywhere, and use all operators in
| prefix-form.
| cmrdporcupine wrote:
| Well there's apparently a not insignificant amount of
| Haskell at the new job I'm starting in a week, so I guess
| I'll see how I fare with it.
| eru wrote:
| Oh, what's the new job?
| int_19h wrote:
| F#, while missing some of the more interesting features of
| OCaml (e.g. the expressiveness of its module system), also has
| indentation-sensitive syntax that ditches a lot of the
| scaffolding, including semicolons in most cases:
|
| https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...
| Lyngbakr wrote:
| They did an episode of Signals and Threads (Jane Street's
| technical podcast) about their build systems back in 2020Y=.
| Highly recommended, as are the other episodes, especially The
| Future of Programming with Richard Eisenbergdeg and What is an
| Operating System? with Anil MadhavapeddySS.
|
| Y=https://signalsandthreads.com/build-systems/
|
| deghttps://signalsandthreads.com/future-of-programming/
|
| SShttps://signalsandthreads.com/what-is-an-operating-system/
| triyambakam wrote:
| Very interesting footnote system. Is it just for fun and
| aesthetic or have a deeper origin or purpose?
| aaronbrethorst wrote:
| whatever it is, it looks like their first time using it.
| here's the last footnote I see from that account:
| https://news.ycombinator.com/item?id=39964109
| Lyngbakr wrote:
| There's no deeper meaning, just done on a whim.
| postexitus wrote:
| For fun and profit.
| robocat wrote:
| the Chicago Manual of Style suggests this order: [?]
| (asterisk) + (dagger) ++ (double dagger) SS (section mark) ||
| (parallels) # (hash)
|
| (Hard to present glyphs that would actually be used in print
| - I've chosen what I think is close - but they are not that
| close because wrong font)
| Majromax wrote:
| Compose key shenanigans can make such typesetting
| significantly easier. That's built-in on Linux/X, but there
| are also easy solutions for Windows systems.
| c0redump wrote:
| Great suggestions, I also really liked "Multicast and the
| Markets"
|
| https://signalsandthreads.com/multicast-and-the-markets/
| rwmj wrote:
| I respectfully have to disagree. As Fedora packager for OCaml
| packages, dune's configuration is yet another one-off, with a
| weird Lisp syntax, hard to understand, difficult to patch. I'd
| far rather people stuck to using more familiar Linux build
| systems. As an upstream OCaml developer, the whole thing falls
| down the minute you need to integrate other programming languages
| into your build (or OCaml code into a code base written in
| another language).
|
| (This rant more or less equally applies to other language-
| specific build systems.)
| UncleOxidant wrote:
| I haven't been doing much OCaml in recent years, but I do
| sometimes check in to see what's going on in OCaml-land, and it
| seems like dune is what everyone is using now - what are the
| alternatives?
| mhitza wrote:
| Using, probably, to be deprecated ocamlbuild, with _tags file
| which is another made up syntax.
|
| But I'm a new OCaml user, and actively starter using
| ocamlbuild because dune added layers of indirection really
| tripped me up at first.
| yawaramin wrote:
| > more familiar Linux build systems
|
| What's a 'familiar Linux build system'? make?
| blobbers wrote:
| They likely mean 'generally used with other languages'. make
| is an example.
| coder543 wrote:
| Make is primarily used with C and C++. It is not commonly
| used in Java, Rust, Go, NodeJS, or hardly anything
| _besides_ C and C++. Make is not "generally used with
| other languages".
| be_erik wrote:
| Make is the common denominator in most projects I come
| across regardless of language. I see lots of frontend
| projects and certainly Go and Rust projects using Make
| quite often.
|
| Ironically many modern C/C++ projects use Cmake to
| generate Makefiles. If anything the inverse of your
| observation is mine.
| coder543 wrote:
| Are those Makefiles doing anything more than calling "go
| build" and "cargo build"?
|
| Because if they're still using the language-specific
| build tools and dependency management systems, then I
| think you would find that the Fedora maintainer higher in
| this thread would not be any happier that there is a
| sugar coating of Make. That's not what they're asking
| for, based on other rants I've seen from Linux distro
| maintainers.
| be_erik wrote:
| The barebones ones do exactly what you mentioned: simple
| calls to the canonical build tool.
|
| The more complex ones at $JOB actually do some caching,
| dependency management, code generation, and compilation.
| dumah wrote:
| Java https://github.com/openjdk/jdk/blob/master/Makefile
|
| Rust https://github.com/rust-
| lang/rust/blob/master/tests/run-make...
|
| Go https://github.com/golang/go/blob/master/src/runtime/M
| akefil...
|
| NodeJS https://github.com/nodejs/node/blob/main/Makefile
|
| OCaml https://github.com/ocaml/ocaml/blob/trunk/Makefile
|
| Python https://github.com/python/cpython/blob/main/Makefi
| le.pre.in
|
| Haskell
| https://github.com/ghc/ghc/blob/master/ghc/Makefile
|
| R https://github.com/wch/r-source/blob/trunk/Makefile.in
|
| CLR https://github.com/mono/mono/blob/main/Makefile.am
|
| Ruby
| https://github.com/ruby/ruby/blob/master/enc/Makefile.in
| coder543 wrote:
| This doesn't prove anything _at all_. Of course the
| toolchain has to be built somehow. Some toolchains use
| make to do that, rather than depending on the previous
| version of the toolchain 's build system. Some toolchains
| are written in a language completely separate from their
| downstream language, so they obviously wouldn't be
| compatible with their own toolchain.
|
| Downstream projects in these languages do not typically
| use Make.
|
| More to the point, I clicked on the Go one, and it's just
| including this tiny "Make.dist" file that does nothing
| except invoke "go tool":
| https://github.com/golang/go/blob/master/src/Make.dist
|
| Wow. So useful.
|
| I clicked on the Rust one, and not only did it seem to be
| specific to some old testing infrastructure, but I found
| this note:
|
| > There are two kinds of run-make tests:
|
| > The new rmake.rs version: this allows run-make tests to
| be written in Rust (with rmake.rs as the main test file).
|
| > The legacy Makefile version: this is what run-make
| tests were written with before support for rmake.rs was
| introduced.
|
| So, it's an obsolete system that is being migrated away
| from.
|
| But, again, the main point is that what the toolchain
| does with its free time has little to do with how end
| user applications are developed, and the complaints in
| this thread were strictly about building applications in
| distros, not about building toolchains.
|
| _If_ an application in one of these languages uses make,
| it is typically just a little syntax sugar around the
| toolchain commands, which does absolutely nothing to
| absolve the project of the complaints Linux distro
| maintainers have about how dependencies are managed.
| tempodox wrote:
| I build my OCaml stuff with `make`. I use `dune` only for
| libraries, because it makes installing them super easy.
| munificent wrote:
| _> more familiar Linux build systems_
|
| "Familiar" is not a property of any system. It's a relation
| between a system and a user.
|
| Some Linux build systems maybe be more familiar to some users,
| but will be less familiar to others. When picking a build
| system, you can't just look at the system itself and declare it
| familiar or not.
|
| It's not even enough to look at the total number of users
| familiar with some thing. Hindi is one of the most familiar
| languages in the world, but you're probably gonna have a bad
| time if you use it for the menu in a cafe in rural Texas.
|
| You have to look at your actual cohort of users (and potential
| future users) and see what's familiar to them. This is one of
| the key reasons why usability is actually a deeply hard
| problem. So much of usability hinges on familiarity, but
| familiarity is a human-specific highly variable property.
| nicoburns wrote:
| As a user of Linux, I wish Linux distributions would abandon
| their build systems which really only work well for C and maybe
| C++, and also only work on one platform (not even the whole of
| linux!) meaning that packaging has to be duplicated for every
| OS, and stuck to using more familiar language-specific build
| systems that work everywhere.
|
| (this rant more or less equally applies to all os-or-distro-
| specific build systems)
|
| ---
|
| This rant is only semi-serious. I do see _some_ value in the
| Linux distribution style packaging. In particular, I do
| sympathise with the need to do cross-language builds. But
| goodness are they a pain to work with, and probably the biggest
| barrier to me shipping software on Linux.
|
| My hope is that eventually an evolution on build systems like
| bazel/buck2 will lead to a truly universal build system that is
| both cross-platform and cross-language. But unfortunately it
| doesn't look like it's coming soon.
| Hasnep wrote:
| If you only care about Linux and MacOS then Nix can be that
| universal cross language build tool.
| mananaysiempre wrote:
| > [Distro-specific build systems are] probably the biggest
| barrier to me shipping software on Linux.
|
| Don't[1]. Ship source tarballs (or VCS tags).
|
| I'll grant that most distros' build systems are antiquated
| and, in places, silly. (That includes Nixpkgs, first released
| 2006.) We could really use some fresh ideas there.
|
| But they're also _not for you_ (or me) in your (or my)
| capacity as a software author. They're there for a person who
| works on packaging software for, most of the time, a single
| distro, and their balance of complexity and flexibility is
| calibrated accordingly. One of the functions of that person
| is also to keep you honest and represent the interests of
| users before you, because they have more expertise than the
| users but not as much of an attachment to your software as
| you. The ecosystem is less healthy when the author tries to
| fill in for the packager.
|
| "But then the users will come to my bugtracker to complain
| about bugs in patched versions!" Pre-Google, we used to have
| a solution for that: a configure option to set the bug
| reporting email, present in all GNU software. Nowadays it's
| not clear what a good solution could be, but it does seem
| like, unfortunately, the author will have to maintain a table
| of packager contact information for the end users.
|
| [1] https://drewdevault.com/2021/09/27/Let-distros-do-their-
| job....
| lmm wrote:
| > This rant more or less equally applies to other language-
| specific build systems.
|
| And yet those language-specific build systems are
| overwhelmingly winning, in pretty much every language.
|
| > As Fedora packager for OCaml packages,...
|
| I honestly think traditional Linux packaging is in the wrong
| here and the problems are essentially self-inflicted (not in
| the sense that individual maintainers are doing something
| wrong, but in the sense that the policy that traditional Linux
| distributions are following is inherently unsustainable. It's
| designed for a pre-CPAN world)
|
| > As an upstream OCaml developer, the whole thing falls down
| the minute you need to integrate other programming languages
| into your build (or OCaml code into a code base written in
| another language).
|
| True up to a point, but frankly the worst case is falling back
| to a terrible C-style build, and "always do terrible C-style
| builds in case you need to integrate with C code" is not a
| proposition that has much appeal.
|
| Much as I wish the whole world would standardise on Maven or
| Cargo, I can't see a realistic path to there without first
| eliminating C, because the C people are never going to agree to
| follow a standard for package repositories.
| poulpy123 wrote:
| > As Fedora packager
|
| The issue is there. Not you or your fellow packagers of course,
| but in the idea that every linux distribution needs it's own
| packaging system, and each version of a distribution it's own
| packages.
| n144q wrote:
| I wonder if anyone has used Jane Street's open source OCaml
| libraries and tools for somewhat serious projects.
|
| I took a brief look at these things, and my impression is that
| their stuff isn't "ready" for anyone outside Jane Street, even
| though they put a lot of effort in building the ecosystem and
| open source their code.
| Kuinox wrote:
| I tried to use their magic-trace tool but I get at least 4
| differents crash when using it on trivial dotnet programs.
| giraffe_lady wrote:
| I do! Their standard lib replacement was used on the project
| where I originally learned ocaml, so I have just always used
| it. You have to be fairly sophisticated at reading fn
| signatures to find things sometimes but otherwise no complaints
| with it. It has some very nice features too for example convert
| functions to/from sexps for all base types can save you a ton
| of time if you know it's there and plan for it.
|
| I've used some of their other libraries too, their logging and
| unit test ppx are common maybe even de facto standards as much
| as the ocaml world has such a thing. I've also used, off the
| top of my head, their code formatter, one of their test
| frameworks, their implementations of some advanced data
| structures.
|
| Sometimes you do run into one like the other commenter said,
| where that shit just _does not work_. It depends on an
| undocumented something they shipped separately, or needs a
| secret bit of config or whatever. These aren 't malicious, I
| open a ticket and come back in a year or two often they'll be
| working.
|
| It's not zero frustration but I appreciate their approach of
| just throwing everything over rather than spending more
| resources testing and polishing fewer releases. Their code
| quality is generally very high and even if I can't get
| something working directly, it provides a rigorous & vetted
| example implementation.
| n144q wrote:
| Thanks! The "depends on an undocumented thing" happens quite
| often for such projects from companies, and is enough to hold
| people back -- in a company where a third party library
| requires approval, such issues mean that a fix could take a
| while before it is available internally, which could delay
| your own work etc. A lot of this probably comes down to the
| ecosystem -- not enough people are using these things to
| discover them early.
| Syzygies wrote:
| I'm running tests for an involved math research project, and to
| my great surprise Lean 4 << Haskell << OCaml in execution times
| (so far).
|
| I love Lean 4, but good luck getting help with it from AI.
| Today's project-in-progress is digesting their reference manual
| to fit well within a 200K context window. We'll see if that
| helps.
| aiono wrote:
| Not a serious project, but I use their stdlib for my personal
| static site generator. I think it has much better API compared
| to the standard stdlib.
| eru wrote:
| We used them at Bloomberg.
| n144q wrote:
| Could you comment on --
|
| How is the design of the APIs? How stable are they?
|
| Does Jane Street respond to bug reports/pull requests (if
| any) quickly?
| eru wrote:
| Alas, I last worked there in late 2017 (or 2016 or so?). I
| don't recall these details.
|
| I worked on Bloomberg DLIB which is basically an
| implementation of
| https://www.cs.tufts.edu/~nr/cs257/archive/simon-peyton-
| jone...
| dgfitz wrote:
| The only thing worse than a complex codebase is a complex build
| system. I think CMake is an abomination. "Hey go learn this new
| language to compile this other language!"
| gpderetta wrote:
| > CMake is an abomination. "Hey go learn this new language to
| compile this other language!"
|
| "go learn this _awful_ new language "
| feznyng wrote:
| Xmake uses lua if you want to give it a try.
| eadmund wrote:
| He did write 'awful,' which is a fair description of the
| experience of writing Lua.
|
| I don't mean that as an insult to Lua's creators. They seem
| like really smart fellows. It's just that the language is
| (to my eyes, with my background) viciously ugly. And
| 1-based arrays, of course, are evil.
|
| It has some neat ideas, though, and it is supposed to be
| _very_ easy to integrate into a project. But man, that
| syntax ...
| wizzwizz4 wrote:
| 1-based arrays are fully optional in Lua. You can use
| -2-indexed arrays instead, should you wish to.
| wbl wrote:
| Ah good old $[ in perl.
| feznyng wrote:
| I kind of agree, but it's still far better than CMake's
| custom language IMO.
| gpderetta wrote:
| I suspect that INTERCAL would be better than CMake's own
| language.
| munificent wrote:
| The only thing worse than a complex build system is a simple
| build system that can't build what you need to build. :(
| dgfitz wrote:
| This means the architecture is also bad, and that is what
| needs to be fixed.
| almostgotcaught wrote:
| so simple huh? since you fancy yourself an architect can
| you let me know how we might fix the architecture of
| https://github.com/llvm/llvm-project such that CMake is no
| longer needed?
| dgfitz wrote:
| Not in a glib response, no.
| almostgotcaught wrote:
| okay can you at least tell me how the architecture of
| https://github.com/llvm/llvm-project is "bad"?
| db48x wrote:
| That's such an obnoxious tactic. It's basically saying
| "Your point is invalid unless you can cut this Gordian
| Knot on the spur of the moment in an internet comment!"
| philipwhiuk wrote:
| It's less bad than the ill informed Dunning-Kruger
| opinion it rebuts.
| dgfitz wrote:
| Don't worry Phillip, ignoring the bit where it isn't a
| rebuttal at all, I know I'm not a smart person. That's
| why I like simple builds!
| db48x wrote:
| It's a logical fallacy; it doesn't rebut anything.
| troad wrote:
| >> Complexity has to live _somewhere_. If you are lucky, it
| lives in well-defined places. In code where you decided a bit
| of complexity should go, in documentation that supports the
| code, in training sessions for your engineers. You give it a
| place without trying to hide all of it. You create ways to
| manage it. You know where to go to meet it when you need it.
| If you 're unlucky and you just tried to pretend complexity
| could be avoided altogether, it has no place to go in this
| world. But it still doesn't stop existing.
|
| >> With nowhere to go, it has to roam everywhere in your
| system, both in your code and in people's heads. And as
| people shift around and leave, our understanding of it
| erodes.
|
| >> _Complexity has to live somewhere._ If you embrace it,
| give it the place it deserves, design your system and
| organisation knowing it exists, and focus on adapting, it
| might just become a strength.
|
| - Fred Hebert, https://ferd.ca/complexity-has-to-live-
| somewhere.html
| lmm wrote:
| I think that's true up to a point, but it misses the
| distinction between accidental and essential complexity.
| Essential complexity has to go somewhere, accidental
| complexity doesn't, and most complexity in software is
| accidental.
|
| And even when the complexity is essential, IMO it's better
| off not in the build system. I'll gladly accept more
| complex code for the sake of a simpler build (even though
| that theoretically means worse performance). Worst case if
| I need to do something complex at build time I'd rather
| model that as "the build system _invokes a program_ that
| does something complex " than try to express the complex
| thing in some Turing Tarpit "configuration" language.
| globular-toast wrote:
| Also the point of software engineering is to take the
| real complexity that actually exists and you can't get
| rid of and work out how to write simple code to deal with
| it. Good software architecture does not include
| complexity because there is complexity in the business.
| It _manages_ the complexity with simple code.
|
| The trouble is when people say "complex" you don't really
| know what they mean, though. They often just mean
| "difficult". Every programmer who wants to use that word
| needs to watch this:
| https://www.youtube.com/watch?v=SxdOUGdseq4
| lmm wrote:
| Nah. Hickey say stuff that sounds profound but doesn't
| actually help you get better at writing software or even
| at talking about it.
| philipwhiuk wrote:
| > accidental complexity doesn't, and most complexity in
| software is accidental.
|
| I strongly disagree. Most software is insufficiently
| complex to adequately represent reality.
| lmm wrote:
| > Most software is insufficiently complex to adequately
| represent reality.
|
| That may be so; what I'm claiming is that most of the
| complexity in software as it currently exists is
| accidental.
| PhilipRoman wrote:
| After seeing enough bitbake recipes, I have to respectfully
| disagree. Some things are better off left unbuilt.
| dgfitz wrote:
| I wish I'd included this in my initial response:
|
| I am painfully aware of bitbake. I've probably written
| 3-400 recipes.
|
| Most of them are about 20-30 lines long, because I refused
| to hide the compilation mess inside a recipe. I fixed the
| problem _before_ getting to the bitbake part. Most of my
| recipes at this point need only a repo name, the recipes
| are identical after that.
| tom_ wrote:
| I don't mind CMake. It's awful, but I'm already using C or C++,
| so the bar is set pretty low. It does do a few things right,
| and at least some of the things it does wrong are manageable.
|
| I'll eat at least a bit of shit if it means I can get more than
| one platform's-worth of build process out of a single set of
| human-editable configuration files.
| queuebert wrote:
| Have you tried meson? I used it once many years ago, and it
| seemed slightly better.
| ahartmetz wrote:
| Meson's approach to simplicity is if you need anything
| nontrivial, you simply add it to Meson itself. In CMake,
| you can script it in more cases.
| homebrewer wrote:
| Depends on what you need; you can easily call out to
| external Python scripts to solve many problems, which
| will work everywhere because Python is needed to run
| meson itself.
| wrngadsa wrote:
| Meson is quite nice if you want to build software exactly
| and exclusively how the Meson designers want to build
| software.
|
| The world does not work like this though. CMake is weird
| but once you've learned some non-intuitive stuff, it works
| very well and there's a reason why pretty much everyone is
| using it.
| foldr wrote:
| I initially tried to use Meson for a small emscripten
| project targeting WASM, but it seemed stuck on the idea
| that I was cross-compiling ( _technically_ true), and this
| seemed to make everything more complicated. With CMake I
| was able to hack together a working build configuration
| without too much trouble. I just had to tell it that I
| wanted to use emscripten as the compiler.
| kevin_thibedeau wrote:
| The ideas embodied in modern target-oriented CMake are
| worthwhile. It's all hobbled by a dreadful script language but
| that could be improved with sufficient motivation.
| tom_ wrote:
| I think what annoys me about it the most is that for every
| good idea in there there's one piece of tedious crap you have
| to work around, one actively bad misfeature that doesn't work
| properly, and one terrible aspect of the awful scripting
| language.
|
| (And two minor annoyances that people determined to hate
| CMake will never shut up about. But then, if you're
| determined to hate something, there are worse things than
| CMake to do it to! So I can't be too critical.)
| eru wrote:
| > I think what annoys me about it the most is that for
| every good idea in there there's one piece of tedious crap
| you have to work around, one actively bad misfeature that
| doesn't work properly, and one terrible aspect of the awful
| scripting language.
|
| Sounds like the perfect match for C++.
| cmrdporcupine wrote:
| It's hobbled not just by that but by the fact that it's not
| the thing doing the building. It's a build system for build
| systems, and suffers from brittleness and confusion as a
| result.
| kevin_thibedeau wrote:
| Make, in all its various implementations, is far more
| brittle than CMake. Ninja requires a meta-build tool like
| CMake.
| yawaramin wrote:
| Dune's crash message:
|
| > I must not segfault. Uncertainty is the mind-killer. Exceptions
| are the little-death that brings total obliteration. I will fully
| express my cases. Execution will pass over me and through me. And
| when it has gone past, I will unwind the stack along its path.
| Where the cases are handled there will be nothing. Only I will
| remain.
| snthpy wrote:
| Love this! Thanks for posting.
| marviel wrote:
| Oh wow it's real
|
| https://github.com/ocaml/dune/issues/2767
| ragnot wrote:
| I feel like I just got initiated into something
| sestep wrote:
| For context, this is adapted from the book _Dune_ , after
| which the OCaml build tool was named.
| https://www.goodreads.com/quotes/2-i-must-not-fear-fear-
| is-t...
| fire_lake wrote:
| What percentage of JS programmers actually work on this stuff I
| wonder?
| piuantiderp wrote:
| The article said the team working on it has 5 engineers.
| philipwhiuk wrote:
| The article says 12.
| thomastraum wrote:
| might get downvoted but all i think of when I see a Jane Street
| article is how much each programmer earns
| eru wrote:
| I hope you are feeling happy for them?
| elzbardico wrote:
| Whenever I see posts from Janestreet blog I laugh inside thinking
| how working on the financial industry at Jane is probably more
| fun and rewarding than some 90% of the jobs in the so-called tech
| industry.
| whydid wrote:
| Grass is always greener my friend.
|
| Many jobs in finance are updating 20 year old Java code, or
| figuring out new ways to load data in and out of Excel files
| for custom reporting.
| ericjmorey wrote:
| Probably not the case at Jane Street
| udev4096 wrote:
| OP should've been more specific. HFT firms, not any other
| finance companies, probably have a lot more exciting work due
| to the nature of reducing latency using all sorts of novel
| techniques.
|
| I wonder if they disable all the fancy exploit mitigation
| protection in linux kernel just for a tiny performance hit
| fuzztester wrote:
| some use a fork of rust with borrow checking disabled.
|
| go fish
| shepherdjerred wrote:
| Isn't borrow checking only compile time?
| SkiFire13 wrote:
| Yes, but sometimes it forces you to write slightly slower
| code that can be proven safe.
| spoiler wrote:
| You don't need s whole compiler for that... You can use
| `unsafe`. Sometimes you don't even need that either!
| francasso wrote:
| Would they also change the aliasing assumptions to
| something close to C/C++? Otherwise I imagine it would be
| relatively easy to make mistakes and get "surprises" at
| runtime thanks to the optimizer.
| n4r9 wrote:
| Mother capitalism deems that our brightest young minds best
| serve humanity in two tasks. Keeping the public passively
| scrolling, and moving money at speed to make wealthy people
| more wealthy.
| udev4096 wrote:
| Absolutely. The moral aspect is certainly questionable.
| Although, I wouldn't say "all brightest minds" are going
| to neglect their moral concerns for getting rich
| eru wrote:
| At one point, Jane Street had a lot of the effective
| altruists flock to them to 'earn to give'.
| jrockway wrote:
| There's also electronic medical records in there
| somewhere.
| deepsun wrote:
| Some fraction of young minds, not all. I was happy to
| work at a small aerospace company with extreme
| concentration of brightest minds. Only because they loved
| the domain, and didn't mind a salary cut. What a joy and
| relief it was for me after FAANG!
| hereonout2 wrote:
| Possibly you didn't mind the salary _because_ you 'd
| already worked at FAANG?
| andrepd wrote:
| > Only because they loved the domain, and didn't mind a
| salary cut.
|
| Yes, so it means GP is right! Modern capitalism in tech
| is about rewarding the two aforementioned tasks.
| vasco wrote:
| Is the problem with the system or with the minds? The
| minds that want to scroll are the same as the ones that
| make money on the scrolling, that made the scrolling
| itself, and that made the system we're in. Why is it that
| defeatist comments always focus on the capitalism part
| and not on anything else? I don't think it's perfect like
| we aren't perfect but unless you have some particular
| suggestion this type of comment just reads "boohoo the
| world is bad and it's not my fault".
| Capricorn2481 wrote:
| > Why is it that defeatist comments always focus on the
| capitalism part and not on anything else
|
| Because endless growth is the only reason these once fun
| spaces have been hyper focused to be as addictive and
| stressful as possible to the "whales" of scrolling.
| That's why, when their own internal reports say "people
| spend unhealthy time on our platform and it's making them
| unhappy," it gets passed up the chain of command and
| whittled down by internal incentives until it dies as an
| issue.
|
| Individuals hold some blame, but to put most of it on
| them is to ignore what growth demands. You're supposed to
| doomscroll and engage and worry. That's the business
| model. Facebook is in the same business as Cigarettes and
| Casinos. When I see someone on an air tank playing slots,
| literally crying when they spend their last dollar, I
| will not waste my breath blaming them. Just like I won't
| blame the doomscroller, anxious that they need to stay
| "informed," who hasn't met the basic needs in their own
| life.
|
| > The minds that want to scroll are the same as the ones
| that make money on the scrolling
|
| No? Where are you getting this? I don't think the people
| guiding these companies want to spend 6 hours scrolling
| TikTok. This is not the way most people live their lives.
| vasco wrote:
| > No? Where are you getting this? I don't think the
| people guiding these companies want to spend 6 hours
| scrolling TikTok. This is not the way most people live
| their lives.
|
| I meant they are all humans. We're all sort of the same.
| If you disagree just think of your opinion of any other
| species, or about a group of people a thousand years ago,
| and you see what I meant.
| thom wrote:
| Hey, some of them are working tirelessly to ruin sports.
| actionfromafar wrote:
| Sports is doing that to itself, but I assume you mean
| betting, which really sucks the fun out of a room.
| amelius wrote:
| > Sports is doing that to itself
|
| No, sponsor contracts, advertisements.
| kitd wrote:
| With the "moving money at speed" part in the highly-
| optimised form of online betting.
| eru wrote:
| Actually, not. These market makers are often prop shops.
| That means they use their own fund (prop = proprietary)
| to do the trading. They can do that because they don't
| need much capital to run.
|
| So the story here is that over the last twenty years they
| stole the lunch from the traditional market makers like
| eg banks.
|
| Of course, they got rich in the process. But they started
| from relatively modest means, compared to the companies
| they took on.
|
| Michael Lewis's 'Flash Boys' is an hilarious account of
| this process. Well, it's involuntarily hilarious, because
| to tell his story, Lewis needs to cast Goldman Sachs (!)
| and other big banks as the victim. See the rebuttal
| 'Flash Boys: Not so fast' by Peter Kovac for more
| insight.
| leg100 wrote:
| None of that is contrary to "moving money at speed to
| make wealthy people more wealthy".
| eru wrote:
| You make new people wealthy.
| n4r9 wrote:
| Perhaps some start this way. But in terms of the general
| trend of talented engineers and mathematicians being
| sucked into this quant vortex, it is a matter of making
| wealthy people wealthier.
| lotsofpulp wrote:
| Automation in trading makes all investors wealthier via
| lower fees. Trading costs basically nothing nowadays, and
| that is because far fewer people are employed to do it.
|
| Obviously, the people who own the automation will want a
| cut of the rewards, like any other business.
| patmorgan23 wrote:
| Automation in trading != HFT algorithms
|
| Obviously NASDAQ and electronic trading systems are a
| good innovation. But firms basically doing arbitrage or
| exploiting uneven network latency are not that
| economically productive.
| kqr wrote:
| Tighter spreads and higher liquidity is not economically
| productive? I can see arguments both ways.
| davesnx wrote:
| this is a good naval-style tweet, well done
| secondcoming wrote:
| Not HFT, but we disable those on our cloud VMs
| deivid wrote:
| Some jobs at HFT deal with a lot of this fun--I was doing
| Linux at one for 6 years.
|
| A lot of jobs are extremely mundane though, compliance,
| regulations, legacy code bases, etc.
|
| Yes, all mitigations get disabled
| cduzz wrote:
| Of course the mitigations get disabled.
|
| What's the threat model where your HFT application's
| running hostile / untrusted code?
|
| I'd like this stuff enabled on my desktop because I'm not
| sure what hideous javascript is being dumped into my
| browser by some advertising network.
|
| But my trading platform? If the badguys are able to
| execute these attacks there, it's because they've got
| full access already.
| nly wrote:
| Most of them simply bypass the Linux kernel altogether
|
| Processes reading and writing directly to FPGA/NIC ring
| buffers.
|
| Shunning TCP in favour of UDP based protocols that are easy
| to optimize for your particular usecas in userspace.
|
| Removing cores from the Linux scheduler entirely and
| pinning processes to those cores.
|
| This stuff isn't even novel, it's been standard practice
| for a couple of decades.
| globular-toast wrote:
| Maybe, but could you live with the fact that you're providing
| absolutely no value to the world? It always makes me sad when
| the greatest minds get gobbled up by these companies because
| they're creaming off the top of an enormously complex system
| they created.
| smabie wrote:
| People outside the industry just use that as cope because
| they making way less money.
|
| "How do you live with yourself making high 6 fig / 7 figs a
| yr?"
|
| Quite easily in fact.
| globular-toast wrote:
| I did work in the industry briefly and can confirm it is
| actually full of people who would post a comment like this.
| Just in case anyone needed another reason to avoid it!
|
| Work on real problems. Try to make real people's lives
| better and happier. There are real problems in finance but
| my feeling was it's all very simple and solved decades ago,
| now it's just pointless complexity that isn't solving
| anyone's problems. I recommend John Kay's _Other People 's
| Money_ for a primer on what finance is actually good for
| and where it's gone wrong.
|
| The real big problem in finance IMO is digital cash.
| Bitcoin started out trying to solve that problem, and there
| are still some people in the community interested in it,
| but it's mostly of interest to the finance guys now. Just
| another "instrument" in their "portfolios".
| philipwhiuk wrote:
| > Work on real problems. Try to make real people's lives
| better and happier. There are real problems in finance
| but my feeling was it's all very simple and solved
| decades ago, now it's just pointless complexity that
| isn't solving anyone's problems. I recommend John Kay's
| Other People's Money for a primer on what finance is
| actually good for and where it's gone wrong.
|
| If the last few years have taught me anything it's that a
| large % of the population will actively aim to make their
| own lives worse long term because they are told lies.
| What benefit is there really in trying to undo their own
| self-inflicted damage.
| cess11 wrote:
| Not trying makes you a worse person.
| vacheron wrote:
| Having worked in both, there is a huge number of really
| smart people who turn down offers to work in quant shops.
| And people really underestimate how much tech will pay top
| researchers/overestimate how much the average quant makes.
| boxed wrote:
| One could argue that the OCaml work they do is skimming off
| the top of the fintech world and putting that money to better
| use :P
| nvarsj wrote:
| As someone that used to work in this world I find this
| opinion of "no value" pretty bizarre. There's a lot of value
| in efficient markets with high liquidity and low fees, which
| companies like JS make possible.
| cess11 wrote:
| They've only been around for a short while so how could you
| know?
|
| Let's see if they last at least a couple of centuries.
| nly wrote:
| I had a friend who worked there and he claims they have a tonne
| of dogshit code and C++ as well.
|
| He also claims they're full of elitists from top universities
| and are not receptive to ideas outside that bubble.
| andrepd wrote:
| Is it? Hours are long and the standards are very demanding, and
| in the end you're just manipulating numbers for profit, so
| that's not (for most people) a very meaningful endeavour.
|
| I still rank it above making people click on ads though.
| vacheron wrote:
| People really over hype and overrate what kind of work happens
| at Jane Street or similarly exclusive HFTs. They have their mix
| of interesting projects and mundane work like any other
| company.
| yodsanklai wrote:
| Anecdote which doesn't prove anything but I have a colleague
| who left JS to work in FAANG after 6 months because working
| there was just too boring in his opinion (despite a higher
| salary).
|
| Also, it's apparently significantly harder to land a position
| at JS than at a Google/Meta.
|
| Actually, I applied there a while ago, the interviewer was
| actually pretty unpleasant, which hasn't happened to me at big
| tech. Didn't leave a really good impression.
| ddellacosta wrote:
| Yeah I sent an application on a whim because I've spent a lot
| of time in FP land professionally and otherwise, I figured
| "what the heck, maybe they'll see value in that," and I was
| surprised to actually get a response but it was dismissive
| and borderline insulting..."there's nothing relevant here"
| talking about my resume is the thing I remember the most.
|
| It was probably the first and only time I would have rather
| have been ghosted lol
| decafninja wrote:
| Jane Street and a handful of few other elite shops make up a
| tiny percentage of the finance industry's tech workforce
| though.
|
| There are a ton more people working in tech in finance that
| don't quite have it as fun (or lucrative) as Jane Street let
| alone your average tech company.
| that_guy_iain wrote:
| "accidentally" built a better build system after setting out to
| build a better build system, it's weird how accidents like that
| happen.
| edg-l wrote:
| i want to like ocaml but the tooling is not there, coming from
| Rust.
|
| The lsp requires you to run "dune build" first, bad already.
|
| If you add a new file, the lsp wont pick it up until you dune
| build it again.
|
| The compiler errors arent there too.
|
| But i loved writing OCaml, its just thats a bit more painful to
| learn than due to the tooling, since i didn't use many functional
| langs before.
___________________________________________________________________
(page generated 2025-01-31 23:01 UTC)