[HN Gopher] Launch HN: Moonrepo (YC W23) - Open-source build system
___________________________________________________________________
Launch HN: Moonrepo (YC W23) - Open-source build system
Hey HN, Miles and James here from Moonrepo (https://moonrepo.dev).
Are you struggling with large codebases? Well look no further! We
built Moonrepo to simplify repository management, project
ownership, task running, and everyday developer and productivity
workflows. If you've used Bazel (or another "enterprise" build
system) in the past, you're probably aware of how complex they can
be to setup, configure, and use. Let alone the cognitive overhead
required by developers on a day to day basis. After more than a
decade in the industry, with many of those years working on
infrastructure and developer tooling related products, we set out
to build Moon, a language agnostic build system. Existing systems
focused solely on runtime logistics (faster builds, concurrency),
while we want to also focus on the developer experience. We do this
by automating workflows as much as possible, in an effort to reduce
manual work. We constantly sync and verify configuration, so that
the repository stays in a healthy state. We also infer/detect as
much as we can from the environment/repository/codebase, so pieces
"just work". We wanted our system to be enjoyable to use and easy
to understand, but also solve the same problems as existing
systems. For example, configuration is in YAML, not a proprietary
syntax. Tasks are defined and run as if you were running them in
the terminal; no more abstractions like BUILD files. Unlike Bazel,
we don't hide or heavily rewrite terminal output, so the feedback
loop is what you expect. We manage a toolchain, ensuring the
correct version of languages is used (no more "works on my
machine"). And lastly, our foundation is built on Rust and Tokio,
so performance is first-class, the runtime is reliable, and memory
safety is guaranteed. We follow the open core model. Moon is open
source, but we're also working on a few subscription-based services
for monitoring and improving your continuous integration pipelines,
a registry of project and code ownership, a continuous
deployment/delivery board, auxiliary application systems, and more.
We haven't finalized the subscription model yet, so there's no
pricing information on the website. However, we do have a
starter/free tier that everyone can use by registering on
https://moonrepo.app. In the future, we will offer on-prem as well.
Although Moonrepo is relatively new, we're already feature-packed,
stable, and used in production. We're big fans of honest feedback,
and look forward to your comments!
Author : mileswjohnson
Score : 118 points
Date : 2023-02-21 18:48 UTC (4 hours ago)
| quickthrower2 wrote:
| Apologies if this sounds reductive but can I think of it as an
| open source CircleCI?
|
| The thing that annoys me most about CircleCI, TravisCI, Github
| Actions and Appveyor is that there is no simple way to run the
| same thing locally to debug or test workflows without creating
| either git history mess or temporary hacks like taking off branch
| restrictions in the yaml.
| angio wrote:
| I'm a big fan of using nix to achieve that.
| https://determinate.systems/posts/nix-github-actions
| KMag wrote:
| I'm not a lawyer, but given that monorepo was in use in the
| versioning/build space long before launch, I would have been
| hesitant to launch with a name where it's going to be an uphill
| battle to enforce trademark.
|
| (I also worked for Google a while back, and they were very
| conscious about not using Google as a verb internally, losing a
| slow battle against the tide of trademark dilution.)
| avarun wrote:
| It took me three readthroughs to notice as well, but the name
| is actually `moonrepo` not `monorepo`.
| tnorthcutt wrote:
| Did you notice that this product is called `moonrepo`, and not
| `monorepo`?
| arnorhs wrote:
| Awesome, congrats. I've been an early trier of moon repo and I
| really fell for the slickness of the website and the name, when
| evaluating build tools.
|
| I've primarily worked in typescript codebases and have used raw
| yarn workspaces, lerna, nx and recently evaluated moon and turbo.
|
| The funky part is I eventually simply went with nx, not just
| because I've used it before, but also because I felt like the
| configuration is just simpler and more lightweight. Esp. since
| you can pretty much roll with it without defining much more than
| single simple config file - while moon required some 2-3 separate
| config files, plus config files in each project (I understand the
| per-project config is not required, I don't remember why I needed
| it - something to do with my build tasks)..
|
| In any case I intend to give it another shot soon.
|
| As for the whole config in yaml vs json vs toml or whatever, not
| a deal breaker since most of the time these sorts of configs are
| perhaps not something you end up interacting with
| programmatically - I know a lot of people enjoy editing yaml more
| than json files
| mileswjohnson wrote:
| Thanks for the feedback. Too much configuration is something we
| keep thinking about, and are working to streamline. It's a bit
| involved since we support multiple languages.
| bunged wrote:
| Why would I want to use this instead of, say, GitLab's CI/CD
| Pipelines or Azure Pipelines?
|
| It's a fair bit of effort to change one's already established
| code repository and build system, and I don't really understand
| from your pitch what you are offering that is sufficiently
| advantageous for a decision maker to choose to switch.
| mileswjohnson wrote:
| So moon wouldn't replace your actual CI pipelines. It would be
| a tool that runs _in your pipeline_ to effectively run tasks as
| fast as possible.
|
| For example, in CI, tasks are only ran if they are affected by
| files changed in the pull request. No more running everything
| unnecessarily. We also support remote caching of artifacts,
| which helps to speed up CI even further, by avoiding long build
| times.
| bunged wrote:
| Interesting, so does it keep all the intermediates from
| previous builds and then does an incremental build on top of
| this? Like doing a local dev build but in the cloud?
| tunkafor wrote:
| We've had a lot of success with Brisk
| (https://brisktest.com/) doing this exact thing. It keeps
| our environment running so we don't have to rebuild on
| every run, kind of like a dev build. It's really fast.
| mileswjohnson wrote:
| Yeah at a high-level, that's how it works. However, the
| incremental caching part does require remote caching of
| artifacts, so that they can be shared across CI jobs/runs.
| bunged wrote:
| Thanks for explaining, that does sound compelling, and
| useful that it can be slotted into an existing system.
| newaccount2021 wrote:
| [dead]
| debacle wrote:
| How are you going to deal with the mass diaspora when your runway
| runs short and VCs want to make money?
|
| This seems to be a cyclical thing with build tools. Things are
| great until you expect customers to pay.
| jpgvm wrote:
| Bazel has a heavy focus on correctness (pretty much to the
| exclusion of everything else). Where does Moon fall on the
| correctness gradient? Does it enforce hermecity or deterministic
| builds or give me tools to accomplish it?
|
| In the same vein as those questions how does caching work? Is it
| content based like Bazel or mtime like Nx et al? If there is no
| sandboxing does it do input tracking or is there manual "cache
| key" shenanigans?
|
| If the configuration language is YAML how am I expected to
| implement new behavior? Is that in Rust? Is there a plugin
| architecture? Do I need to rebuild the tool itself and
| redistribute it to my build machines and developers? The main
| appeal of Starlark/Python in build systems is ability to create
| macros or in many cases define entirely new rulesets to support
| new tools and languages without needing to modify the complex and
| performance sensitive core.
|
| Sorry for the skeptisicm but build systems are very complex
| beasts and new entrants like Nx don't measure up to tools like
| Bazel very well.
| mileswjohnson wrote:
| > Does it enforce hermecity or deterministic builds or give me
| tools to accomplish it?
|
| I wouldn't say moon is hermetic, nor are we trying to be. We
| don't use the sandbox approach for tasks, and run against the
| original files. For the languages we support, this works best.
|
| As for deterministic, we try to be. We have a "toolchain" where
| we download languages/tools in the background, and run tasks
| using these tools. This ensures, at minimum, that the same
| version/variant of a tool is used across machines.
|
| > In the same vein as those questions how does caching work?
|
| It's content based. We also hash other kinds of inputs
| depending on the language that is running.
|
| > If the configuration language is YAML how am I expected to
| implement new behavior? (and other questions)
|
| Our focus right now is on non-compiled languages, primarily web
| languages, where custom behavior (like Starlark) is not
| necessary. In the future, this may change.
| andrew_ wrote:
| Bazel as a taskrunner/build system is to most monorepos as a
| Pile Driver [1] is to a floorboard nail. It will do
| _everything_ but it 's massive overkill. We are agreed that Nx
| doesn't measure up, but I wouldn't compare it to Bazel.
|
| [1] https://en.wikipedia.org/wiki/Pile_driver
| rfoo wrote:
| Fair point. I treated the product as a build system and it
| just doesn't look right. However it makes perfect sense if it
| advertised itself as a "task runner".
| thundergolfer wrote:
| To be clear, Bazel focuses on correctness because it's
| _essential_ to acheiving performance at scale.
|
| If you don't have correct caching of intermediate build
| artifacts, a system can't handle the compile and test
| requirements of large codebases.
| pdeva1 wrote:
| one of the reasons Bazel needs BUILD files with explicit inputs
| /outputs defined per file is to do fine grained incremental
| builds and test runs. so if i change say foo.c, i only need to
| recompile foo.obj and run 'foo-tests'. Moon seems to take globs
| as input. Thus modifying even a single file inside 'src' dir will
| trigger rebuild/retest of the entire 'project'
| mileswjohnson wrote:
| Our configuration is using globs but under the hood we content
| hash all the files that match the glob, and only run/cache if
| the aggregated hash has changed.
|
| For the languages we currently support, this is more than
| enough. Once we dive deeper into compiled languages (probably
| starting with Rust), we'll look into more granular reactivity
| and possible use something like sccache.
| thundergolfer wrote:
| Bazel's glob does the same thing. I'm not sure what the OP
| means, Bazel's incrementality is aided by fine-grained build
| input specification, but it's incrementaility is at core a
| combination of deterministic build rules, storage of build
| rule execution history, and early-stopping.
| kccqzy wrote:
| I don't see how that solves the problem mentioned by OP. If
| the build rule mentions foo.c, we only need to recompile that
| one object and relink. When you are using globs, changing one
| file changes the aggregated hash and then would necessitate
| recompiling every object file.
| ctvo wrote:
| How do you compare with BuildBuddy? BuildBuddy embraces Bazel,
| but makes it easier to setup and operate Bazel when you don't
| have Google infrastructure.
|
| - https://www.buildbuddy.io/
|
| - https://github.com/buildbuddy-io/buildbuddy
| mileswjohnson wrote:
| We have a long ways to go, but our end goal for moonbase is
| basically buildbuddy, but for non-bazel users. Right now
| moonbase requires moon, but we're looking to decouple it.
| beisner wrote:
| What's the motivation for using YAML instead of Starlark (Bazel,
| Buck) or something closer to Python (Pants, please.build)? Seems
| as though much of the other monorepo tools have (kinda)
| standardized on this.
|
| As I understand it, the primary reason these build systems
| leverage these Python-variants is so that the build rules,
| toolchains, constraints, and build definitions can all be written
| in the same language (since build rules often require some
| programmatic behavior). Perhaps with a future vision of them
| being totally interoperable across build systems.
| Denzel wrote:
| Not to mention using an actual language aids readability,
| extensibility, and static analysis, unlike a data exchange
| format. Starlark is a benefit, not a con, so YAML feels like a
| major step backwards.
|
| I'm generally happy with Blaze/Bazel, so I'm not necessarily in
| the target market for Moonrepo, I guess.
|
| EDIT: This isn't really competing with Blaze/Bazel either when
| I look at the execution model. It goes back to imperatively
| defined tasks instead of declaratively defined dependencies,
| which feels more spirtually aligned with Make than Blaze/Bazel.
| mileswjohnson wrote:
| Yeah that's fair feedback. At this time, we consider
| ourselves more of a Bazel-lite than an actual Bazel
| replacement. Once we support more languages and features,
| this may change in the future.
| Denzel wrote:
| I'm not trying to be overly critical here, in fact, I want
| to share the type of empathetic feedback I'd hope to
| receive as a founder.
|
| Have you actually talked to Blaze/Bazel users to understand
| what frustrations (if any) they have with their current
| build system? Have these users asked for a Bazel-lite? If
| so, and you still want to position yourself as Bazel-lite,
| then you should include some of their direct feedback and
| write your messaging accordingly.
|
| As a daily Blaze/Bazel user, I don't have a desire for a
| Bazel-lite. I've worked at a midsize company that used
| Bazel, and I'm working at the company that created
| Blaze/Bazel.
|
| Disclaimer: Opinions expressed are my own; not
| representative of my employer.
| howinteresting wrote:
| I think it's more illuminating to talk to people who
| tried to use Bazel and failed, often wasting months in
| the process.
| mileswjohnson wrote:
| A lot of our decisions were based on our past experience
| with Bazel. We both have worked at companies that tried
| to use Bazel and failed miserably. There's stuff we like
| about Bazel (and copied in some capacity, like file
| groups), and definitely parts we hate about Bazel.
|
| Bazel doesn't work well for every language, but for those
| languages that it does, it definitely makes sense to use
| Bazel. For those languages that work better with
| something more lightweight, that's where moon comes in.
| I'm assuming you work at Google, so you're experience
| around Bazel is probably much better than those that
| don't work at Google.
|
| I appreciate all the feedback and comments though, very
| much appreciated.
| rfoo wrote:
| I see that you are targetting the Web ecosystem. IMO
| using Bazel in this case was pretty tough, and may still
| be. The way Bazel does things may not make sense for 95%
| JS projects (and 99.9% FOSS projects), so I wonder
| whether it's better to simply drop the mention of Bazel.
| Just don't compare to Bazel.
|
| Better "monorepo-ish" (whatever that means in the
| frontend world) tools are still valuable, but as GP said,
| it's pretty crowded here.
|
| I don't know if you have talked with people working on
| C++ or Java projects (I see that these are not among your
| supported languages), but if you do, you get to see why
| people are talking about Bazel.
| mileswjohnson wrote:
| Yeah, we kind of regret mentioning Bazel in the original
| post, but we can't change it now.
| ctvo wrote:
| Let me give them feedback in the vein of what you're
| asking for. I want the following as a user:
|
| - Distributed cache of built artifacts. Work with my
| company's network topology and security requirements.
| Make this easy to setup, operate, and seamless for
| developers to opt into
|
| - Seamless monorepo support (multiple languages)
|
| That could be considered Bazel-lite. And yes, I'd take it
| over Bazel currently.
|
| If you don't work at Google, convincing your engineering
| organization into learning Bazel is almost always a non-
| starter. Who uses Bazel in the wild? Xooglers primarily.
|
| Their value proposition is sound.
| mileswjohnson wrote:
| Thank you. This is how we feel as well.
| Denzel wrote:
| Off the top of my head, some companies that successfully
| use Bazel: SpaceX [1], Datadog [2], Databricks [3],
| Stripe [4], Uber [5], etc. That's a good 10k+ engineers
| using Bazel successfully in the wild. There's more, of
| course.
|
| [1]: https://www.youtube.com/watch?v=t_3bckhV_YI
|
| [2]: https://www.youtube.com/watch?v=H67uuwVO1tc
|
| [3]: https://www.databricks.com/blog/2019/02/27/speedy-
| scala-buil...
|
| [4]: https://stripe.com/blog/fast-secure-builds-choose-
| two
|
| [5]: https://www.uber.com/en-SE/blog/go-monorepo-bazel/
| rcme wrote:
| > Not to mention using an actual language aids readability,
| extensibility, and static analysis, unlike a data exchange
| format.
|
| Clearly, you've never used Gradle.
| andrew_ wrote:
| I much prefer YAML rather than having to write more code for a
| build system.
| mileswjohnson wrote:
| We chose YAML for a few reasons. The first being that we wanted
| a format that is basically universally supported everywhere.
| This filtered down to JSON, TOML, and YAML. JSON is an awful
| configuration format, so that was a no go. TOML is pretty
| great, but is also not very ubiquitous. That left us with YAML.
|
| The second reason is we wanted something language agnostic and
| not proprietary. It also helps that many other tools, like
| GitHub actions, are written in YAML.
|
| And lastly, we wanted a format that developers are familiar
| with, and won't need to spend time learning. Requiring
| developers to understand a special syntax simply to define a
| task to run is something we want to avoid.
| beisner wrote:
| Gotcha, understood. Mirroring @Denzel's edit, I think that
| rationale makes sense for configuring some of the
| intermediate glue which is project-specific. For instance,
| running test suites, orchestrating deployment, and other more
| custom actions which aren't terribly reusable across projects
| (think Bazel's 'genrule'). Caching test results in a
| distributed way is a great use case, and environment
| isolation (i.e. without forcing the user to define a
| Dockerfile or similar, and leaving environment configuration
| to the build system) seem like great use cases.
|
| However, I think for describing actual build toolchains this
| way of implementing things might end up being significantly
| more complicated (and require even more arcane end-user
| cognitive load) than the Starlark/Python-based build rule /
| toolchain / constraint approach for actually assembling
| libraries, binaries, and other compiled artifacts. There are
| a combinatorial number of backends, conditional options
| settings, etc. which will be hard to capture with a purely
| declarative system. For instance, a C++ binary might needs
| platform-specific compiler flags, or some #ifdef nonsense.
| YAML doesn't have a clean way of implementing conditionals
| based on some constraint. So for heterogenous ecosystems
| (C/C++, Python, container assembly, GPU development,
| microcontrollers, etc.) this pushes a ton of complexity into
| the build rules themselves which may be opaque to end users
| (and thus introduce a ton of cognitive load, steep learning
| curves, hard-to-debug errors, etc.).
| mileswjohnson wrote:
| I don't disagree here. At the moment, moon leans closer to
| a task runner, with hashing, incremental caching, and other
| features tacked on. All of our currently supported
| languages are not compile based languages, and run with
| pre-built binaries.
|
| Once we starts supporting more languages, especially
| compile based ones (probably Rust first), our decision
| around YAML will probably change. Off the top of my head, a
| non-YAML format would probably be used as a secondary
| format, kind of like how a Dockerfile works.
| dmitriid wrote:
| > The second reason is we wanted something language agnostic
| and not proprietary.
|
| You will end up developing your own yaml-based DSL that is
| incompatible with everything else, has weird limitations and
| poorly documented constraints... just like any other YAML-
| based system.
|
| And by the time you'll have written a 1000-line YAML (where
| half of the file is nothing more than awkwardly quoted and
| escaped bash scripts) and realise that YAML cannot be split
| into reusable files, it will already be too late.
|
| Oh. I just saw that you already use a custom non-standard
| extension of YAML with `extends` so there you are
| randall wrote:
| IDK. I've been using buck for a bit, and having python within
| grasp is pretty useful. I don't like yaml for a variety of
| reasons, but the biggest of which is I like my config being
| able to be generated... maybe that's an anti pattern? IDK.
| Would love to know more about your take on yaml vs anything
| more pythoney.
| computershit wrote:
| What makes yaml not generatable?
| randall wrote:
| Sorry, I meant more dynamic than generatable... like
| doing stuff in the build file to build something complex
| in a more simple way.
|
| It's really 'do you make config an artifact, or do you
| make it code' sort of argument.
| mileswjohnson wrote:
| I'm not very familiar with Buck. But I'm assuming it works
| the same way as Bazel's Starlark?
|
| I can see the benefits of "generating code" within the
| BUILD file, but honestly, we haven't required it yet, and
| have been able to do most things with explicit
| configuration. Our token syntax helps with some dynamic
| aspects of this.
|
| Maybe in the future we'll revisit this.
| QuiiBz wrote:
| Congrats on the launch! I've been following Moon since a few
| months, seems like an interesting project.
|
| Could you explain why any existing project using Turborepo/Nx
| should switch to Moonrepo? What are the advantages and
| disadvantages? The support for multiple languages seems like a
| big advantage.
| mileswjohnson wrote:
| I can speak to both of these.
|
| I'll start with Turborepo. Turbo is primarily a task runner for
| `package.json` scripts with some caching... and that's
| basically it. If that's all you need, then great, but if you're
| looking for more functionality, that's where moon comes in.
| moon is more than just a task runner, we're aiming to be a
| repository management tool as a whole. This includes
| project/code ownership, direct CI support, future CD support,
| code generation, hooks management, constraints, release
| workflows, and much more. With that being said, we do have a
| comparison article against Turbo:
| https://moonrepo.dev/docs/comparison#turborepo
|
| As for Nx, they're more of a competitor than Turborepo. Nx and
| moon are aiming to solve the same problems, but go about it in
| different ways. Nx is Node.js based and requires heavy adoption
| of their ecosystem (@nrwl packages) and their executors
| pattern. In the long run, this becomes a heavy source of tech
| debt, as your dependencies are now tightly coupled to their
| packages and release timelines. With moon, we wanted to avoid
| this all together. There are no coupled dependencies, and tasks
| are ran as if you ran them yourself on the command line. No
| abstraction layer necessary. We also want to embrace a
| language's ecosystem as much as possible, so moon adoption
| should be rather simple and transparent (at most each project
| has a moon.yml file).
|
| But to your last point, we agree, multi-language support is a
| massive advantage. Having both backend and frontend code in the
| same repository, powered by the same build system, is a massive
| win in maintenance costs and developer time saved.
| QuiiBz wrote:
| Thanks for your detailed answer.
|
| > release workflows
|
| Looking forward for this, especially if that also means auto-
| publishing of NPM packages, Rust crates, etc.
| mileswjohnson wrote:
| Yup exactly that! We want a single to to handle version
| bumping, changelog generation, publishing to a registry,
| etc, for _all_ languages that we support.
| [deleted]
| aidanhs wrote:
| (for context - I'm not interested in first class node support)
|
| This seems pretty cool. I particularly like how 'gradual' it
| seems to be relative to things like Bazel, i.e. you can take some
| shell scripts and migrate things over. I did have a play and hit
| an initial problem around project caching I think, which I raised
| at [0].
|
| One comment, from the paranoid point of view of someone who has
| built distributed caching build systems before is that your
| caching is very pessimistic! I understand _why_ you hash outputs
| by default (as well as inputs), but I think that will massively
| reduce hit rate a lot of the time when it may not be necessary? I
| raised [1].
|
| Edit: for any future readers, I spotted an additional issue
| around the cache not being pessimistic enough [3]
|
| As an aside, I do wish build systems moved beyond the 'file-
| based' approach to inputs/outputs to something more
| abstract/extensible. For example, when creating docker images I'd
| prefer to define an extension that informs the build system of
| the docker image hash, rather than create marker files on disk
| (the same is true of initiating rebuilds on environment variable
| change, which I see moon has some limited support for). It just
| feels like language agnostic build systems saw the file-based
| nature of Make and said 'good enough for us' (honorable mention
| to Shake, which is an exception [2]).
|
| [0] https://github.com/moonrepo/moon/issues/637
|
| [1] https://github.com/moonrepo/moon/issues/638
|
| [2] https://shakebuild.com/why#expresses-many-types-of-build-
| rul...
|
| [3] https://github.com/moonrepo/moon/issues/640
| mileswjohnson wrote:
| Thanks for the feedback and prototyping with it immediately!
| Always appreciated to get hands on feedback.
| pplonski86 wrote:
| Congrats on launch! Do you have examples of monorepos, for
| example React with TypeScript and Django with Python?
| mileswjohnson wrote:
| Thank you! We have an example monorepo
| (https://github.com/moonrepo/examples) but at this time it's
| only JavaScript. We're working on adding Go to this repo, but
| we personally don't have enough Python experience to add
| Python. Always open to contributions!
| IceWreck wrote:
| You're building something like buck/blaze and replacing starlark
| with YAML ? I want a full programming language when defining
| complex rules. Build systems do a lot more than execute commands.
|
| You lost me there. Buck/Blaze aren't perfect but configuration
| was never an issue.
|
| Moon like it is currently is a glorified task runner.
| quickthrower2 wrote:
| We started using https://nuke.build/. Early days so can't
| comment too much but it seems good so far.
| bovermyer wrote:
| Would this be overkill for a personal project that is currently
| two repos, but I'm having to copy-paste code from one to the
| other whenever I make a change to a particular section?
| mileswjohnson wrote:
| Can you talk a bit more about what kind of code is being
| copied?
| bovermyer wrote:
| The main repository is a single page app that's mostly
| Typescript compiled via Svelte down to a JavaScript bundle.
|
| Most of that repository is a series of modules in their own
| directories (e.g., "src/modules/character").
|
| The second repository is a Node.js API. The majority of its
| functionality comes from one of the modules from the first
| repository that I copy in its entirety to the other
| repository whenever I change it.
| mileswjohnson wrote:
| Gotcha. While moon doesn't solve this directly, you do have
| a few options here:
|
| - Use git submodules. Have the node repo have a submodule
| on the app repo.
|
| - Publish the shared code to a private registry (like
| github's package registry), and pull it in as an npm
| package.
| Aeolun wrote:
| Could you add the character module as a new package, the
| import it in the server with something like "char-package":
| "file:../../client/modules/character"?
| iskela wrote:
| You should check git submodules out. Basically you could
| have your api-repo checkout certain commit from the spa-
| repo to a folder in the project root and reference the
| module reltive to that path.
| jcalder wrote:
| > We wanted our system to be enjoyable to use and easy to
| understand, but also solve the same problems as existing systems.
| For example, configuration is in YAML, not a proprietary syntax
|
| I'm incredibly skeptical of this.
|
| I'm ex-meta and have worked a lot with the enterprise solutions
| you're talking about and the choice of starlark (originally
| python) as the build definition language is one of the killer
| features of the systems.
|
| People want to create macros, target generators, etc. It's a
| common use case for a lot of engineers and IMO is a pretty killer
| feature.
|
| Being able to say "This is an X86 Python Binary", "This is an M1
| python binary" and then bundle those into "this is how you build
| either of those binaries based on inputs" without ever touching
| the internals or anything other than (more or less) a blob of
| python is why those tools scale organizationally.
|
| It allows the teams that need to do weird stuff to unblock
| themselves without drowning the tools org. Sure, it has draw
| backs. Super deep macro layers are kinda a crime against humanity
| and debugging/evolving them can be quite expensive, but I think
| that's just the cost of software.
|
| If that logic isn't in the build definitions it'll expand into a
| meta layer that generates configuration (I've seen giant
| "Translate this definition into 30 configs to run stuff" systems
| time and time again).
|
| I may just be super biased from past mistakes and wins, but I
| think what you're doing is just moving the complexity out of your
| tool into neighboring tools and selling it as a win isn't really
| true, it's shuffling complexity around not removing it.
| mileswjohnson wrote:
| Based on everyone's feedback about YAML (we didn't expect this
| much), we'll probably reconsider this!
| gen220 wrote:
| Take a look at Apache Aurora [1] if you want some inspiration
| on how to mold python into a config language. I used this
| system for a few years and mainly agree with the person
| you're replying to - having a proper programming language to
| define config is a very nice feature that I miss.
|
| [1]: https://aurora.apache.org/
| mileswjohnson wrote:
| Thanks will take a look at.
| jcalder wrote:
| Glad to hear it!
|
| To synthesis my comment down (because it's easier once I've
| written it once, poorly):
|
| The complexity you see in Bazel/Buck/Pants build files may
| seem like a result of their decision to use a programming
| language. That's a red herring. That complexity is
| fundamental to the problems people need to solve to build
| software. If you remove the ability to solve it in your build
| system the complexity will move to other systems.
|
| Wrappers, CI, whatever. The complexity is just the problem
| space rather than a result of bad tooling.
| jedberg wrote:
| My advice to anyone making a new build system:
|
| Most likely you did this because you felt all the other ones are
| too complicated.
|
| But the reason the "enterprise" ones are so complicated is to
| serve their enterprise customers, who need "just this one
| feature" so they can use it. But those customers pay the bills.
|
| So basically you have to choose complexity and profit or
| simplicity and less (or no) profit.
|
| Good luck! But make sure you're ready to make that choice.
| kkarimi wrote:
| I reviewed the market for JS monorepo tools a few months back and
| found NX to be a strong choice. Looking at Moon it seems like the
| syntax is quite nice but I don't see graph feature of NX? Can
| moon only run on affected packages?
| mileswjohnson wrote:
| Yes it can! That's pretty much the only way it works. The `moon
| run` command will only run if affected by changed files, and
| `moon ci` will only run affected tasks/projects in CI
| pipelines.
| haberman wrote:
| > Tasks are defined and run as if you were running them in the
| terminal; no more abstractions like BUILD files.
|
| But abstractions are a good thing! They let you separate a system
| into layers, so that things programmed at a higher layer don't
| need to know all the details of the lower layers. It's the way we
| separate interface from implementation.
|
| Let me give an example from Bazel. Suppose you are compiling some
| C or C++, and you need to define a preprocessor symbol. There are
| two ways of doing this, "defines" and "local_defines":
| cc_library( name = "my_lib", srcs =
| ["my_lib.cc"], hdrs = ["my_lib.h"], #
| This will pass -DFOO when compiling this library #
| *or* any library that depends on it. defines =
| ["FOO"], ) cc_library( name =
| "my_lib", srcs = ["my_lib.cc"], hdrs =
| ["my_lib.h"], # This will pass -DFOO when compiling
| this library # only. Libraries that depend on us
| will *not* get # the symbol.
| local_defines = ["FOO"], )
|
| You can use "defines" when you have #ifdef statements in headers
| and "local_defines" when you only test the symbols in your .cc
| files.
|
| I did not have to think about how defines will propagate to the
| libraries that depend on me, I just specified what I need and the
| build system handles the rest. I did not have to define my own
| CXXFLAGS variable and handle concatenating all of the right
| things together. The lower layer takes care of that.
|
| What Bazel lets you do is create abstraction boundaries within
| the build system. People can write rules that define a self-
| contained set of build logic, then other people can use those
| abstractions without knowing all the details of how they are
| implemented.
|
| Bazel is not perfect, and I have found myself banging my head
| against the wall many times. But overall the ability to define
| abstractions in a build system is, I think, one of the biggest
| things it gets right.
|
| Disclosure: I work for Google, but not on Bazel.
| Aeolun wrote:
| Maybe that's more relevant for the non-js ecosystem? I never
| find myself wanting to modify my source while it is being
| built.
| theamk wrote:
| Look at https://moonrepo.dev/docs#supported-languages
|
| They only fully support Javascript. The complex stuff, like
| defines, C++ toolchain, dynamic libs, etc.. is all out of
| scope.
| jcalder wrote:
| Also, MIGRATIONS!
|
| Interfaces are VERY GOOD for migrations.
|
| If you decide that you want to stop using some compiler flag,
| or maybe use a different compiler, or change your python
| version, or...
|
| You can right a regex to go over all your shell invocations,
| change them, then test, or you can do something like:
|
| ``` def build_thing(name, srcs, hdrs, migration=False): if
| migration: ...
|
| build_thing( name = "my_lib", srcs = ["my_lib.cc"], hdrs =
| ["my_lib.h"], ) ```
|
| You can then manually flip that to test stuff, write a script
| to flip it for every team, sending out a PR (because your
| targets can have oncalls) and they can land it, and then at the
| end you can flip the default and manually add a False to the
| holdouts.
|
| All this stuff gives you the ability to do hard stuff at scale.
| djstein wrote:
| congrats on the launch! can you share any information on how the
| system is currently deployed? or how it will be deployed for on
| premises solutions?
|
| also: are you looking for any other founders?
| mileswjohnson wrote:
| We haven't built the on-prem solution yet, but we're leaning
| towards using Helm charts + Kubernetes. At minimum, it will all
| be Dockerfile based.
| joeswartz wrote:
| Congrats on the launch! Indeed interesting space. I wonder if it
| has some overlap with https://skaffold.dev/. I would be curious
| to hear the differences.
| lhorie wrote:
| IMHO, if you're targeting the Javascript ecosystem, this area is
| already fairly crowded, with Turborepo, Nx and various open
| source tools providing various degrees of functionality (Bazel,
| Pants, Lerna, etc) already competing in the space.
|
| I'm a tech lead for the web monorepo at Uber. We talked to the
| Turborepo guy a few years ago, and he admitted that he wasn't
| sure if it could handle our scale in terms of all the bells and
| whistles that a repo of our size leverages - and his is one of
| the more feature packed commercial offerings in this space.
|
| As a random example: we see thousands of commits a week, so
| invalidating the whole build graph when the lockfile changes is a
| complete non-starter, yet most turn-key solutions target small
| teams and are not equipped to cope with this problem. Phantom
| dependencies[0] are a problem with disjointed build graphs. Etc.
|
| As someone who's evaluated a ton of these systems, I would love
| to see where your experience in this space is coming from. A new
| kid in the block will have a lot to prove.
|
| [0] https://rushjs.io/pages/advanced/phantom_deps/
| mileswjohnson wrote:
| We agree. We weren't happy with all of the current solutions,
| at least in the JavaScript space.
|
| In regards to build graph invalidation, we attempt to hash
| pieces at the granular level. This includes per file content
| hashing, and for dependencies (those in `package.json`), we
| parse the lockfile and extract the resolved version/integrity
| hashes. We can probably improve this further, but this has been
| working great so far.
|
| As for phantom dependencies, this isn't something moon solves
| directly. We encourage users to use pnpm or yarn 3, where these
| phantom problems are less likely. We don't believe moon should
| solve this, and instead, the package managers should.
|
| If you have any other questions or concerns, would love to hear
| them.
| lhorie wrote:
| I guess I'll echo some of the other comments re:
| extensibility. For us, Bazel's Starlark language is useful
| because it allows us to integrate w/ non-JS things like
| protobuf schema definition files. I suspect most companies
| are going to have hybrid stacks instead of Node.js-only ones.
| FWIW, Bazel vs Buck have some history wrt adoption wars, and
| Bazel largely pulled ahead due to a better ability to
| flourish an ecosystem (hell, the entire JS ecosystem around
| it is done via the extensibility system). It very much
| reminds me of how Typescript won against Flow.
| palmdeezy wrote:
| Turborepo author here...
|
| We do not invalidate the whole graph anymore for a lockfile
| change. We now have a sophisticated parser that can calculate
| if a change within the lockfile should actually alter the hash
| of a given target. In addition to higher cache hit rates, this
| is what powers our `turbo prune` command which allows teams to
| create slices of their monorepo for a target and its
| dependencies...useful for those building in Docker.
|
| Prune docs: https://turbo.build/repo/docs/reference/command-
| line-referen...
|
| Turborepo is much more scalable now than when we spoke pre-
| Vercel acquisition. It now powers the core web codebases at
| Netflix, Snap, Disney Streaming, Hearst, Plex and thousands of
| other high-performance teams. You can see a full list of users
| here: https://turbo.build/showcase
|
| Would be happy to reconnect about Uber's web monorepo sometime.
| [deleted]
| renke1 wrote:
| Can Prune be used to build a bundle (as in a zip) for, say
| AWS Lambda, which includes only the dependencies (and not dev
| dependencies)? I've played around with pnpm's deploy but it
| felt a bit lackluster. Especially talking about situation
| where one has a backend package and some shared package. The
| bundle should contain all dependencies (but not dev
| dependencies) of the backend package and shared package and
| of course the built shared package should also be include in
| the bundle's node_modules.
| palmdeezy wrote:
| Yes! Prune then zip the output folder.
| andrew_ wrote:
| Nx is hot garbage imo. Buggy, overly verbose, inflexible,
| poorly documented, and I'm dubious about their peer review
| process.
|
| If I don't need build caching I'm not using any tool but PNPM
| and its workspace toolset - that's literally all most people
| need for a monorepo. I've looked into Turborepo, and its
| simplicity versus Nx is a strength. However, it's not the
| taskrunner that I want.
|
| I now work in a monorepo where build caching is required, so
| I'm excited about moon and keeping a watchful eye on the
| project's progress. From my evaluation so far, it fixes all of
| my gripes about Nx and I'm keen on it not trying to do too
| much, while allowing me to make it as flexible and extensible
| as I need. Extending configs is _chefs kiss_
| mileswjohnson wrote:
| If you end up using moon, and there's anything
| abrasive/convoluted, let us know! We're always looking for
| ways to streamline.
| duped wrote:
| I'm supremely disappointed to see another service using YAML to
| configure task running. I do in fact need a real programming
| language to do this, and copying others in this vein is
| inheriting mistakes and not picking a battle-tested solution.
|
| What you will find is the vast majority of your configurations
| will invoke a make.sh script that does everything that you want
| to support in your system.
| Aeolun wrote:
| Yeah, I'm kind of aligned with this. Github workflow
| definitions would be much more pleasant to write if they were
| plain Typescript.
|
| Feel like any data definition language eventually bolts on a
| shitty form of templates.
| andrew_ wrote:
| Conversely, if moon required code instead of config, I wouldn't
| give it a second look. Ecosystem needs vary greatly.
| mileswjohnson wrote:
| Can you speak to what kind of "functionality" you need a
| language for? Are you referring to Starlark-like files?
| duped wrote:
| Turing completeness. No, I'm referring to using Python (waf,
| conan, etc), Javascript (nodejs scripts, esbuild, etc), and
| other "real" programming languages.
|
| It's incredibly frustrating to have to configure a build with
| something as shitty as YAML. There's a tangible amount of
| money I have wasted in organizations on it.
|
| A build is not actually a static configuration of another
| system. It's a program and deserves everything we need from
| programming languages.
| Iv wrote:
| That sounds like a "there are too many competing standards, let's
| make standard to unify them!" situation.
| mileswjohnson wrote:
| _insert xkcd comic_
___________________________________________________________________
(page generated 2023-02-21 23:00 UTC)