[HN Gopher] Josh: Get the advantages of a monorepo with multirep...
___________________________________________________________________
Josh: Get the advantages of a monorepo with multirepo setups
Author : oftenwrong
Score : 201 points
Date : 2021-07-15 13:25 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| [deleted]
| Meleagris wrote:
| This seems to provide the same functionality as using Git Sub
| Modules[1].
|
| Am I getting the correct impression?
|
| [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules
| Dobbs wrote:
| Assuming I understand what this is doing correctly, it does the
| reverse of a submodule.
|
| git submodules let you tack a second git repo onto an existing
| one. For example repoA tracks `repoB@version1234` at path
| `/foo/bar/baz`.
|
| This on the other hand lets you take monorepo and checkout
| `/go/mysubservice` as a "repo" and treat it as its own repo.
| Then when you do git pushes etc, it translates the changes into
| the larger monorepo.
| creamytaco wrote:
| I don't see what problems this solves, but I do see plenty of
| problems it could introduce. There are tremendous benefits to
| sticking with the tried and tested approach supported by git
| rather than introducing yet more tooling.
| codetrotter wrote:
| From the title I expected it to be a tool for treating multiple
| separate repos as though they were all just one single monorepo.
| But from the description in the README, it seems to be for
| treating subsets of a monorepo as though they were separate
| repositories.
|
| PS: The title, in case it is changed, is currently "josh: Get the
| advantages of a monorepo with multirepo setups"
| tusharsadhwani wrote:
| You might want to check out https://github.com/asottile/all-
| repos :)
| oftenwrong wrote:
| I apologise for the title. HN has a short limit for title
| length, so I came up with my own title. I thought this title
| did a decent job of presenting, using short language, the main
| application that the authors gave top-billing in the README.
|
| I am not affiliated with the project.
|
| JOSH claims to be reversible, so it could be used in either
| direction, which is where the multiple use cases come in.
| Treating subsets of a repo as their own repo, or treating
| multiple repos as one. I would say there is some application
| overlap between this and git submodule/subtree/subrepo and also
| tools like copybara.
| lbhdc wrote:
| That was what I expected from the description as well. After
| reading the readme it's not clear to me what problem this is
| trying to solve, and why this is the solution.
| ganafagol wrote:
| The problem is that large codebases tend to have a huge
| footprint if you need to clone the whole repo. Git as-is does
| not allow you to only pull a subset, i.e. specific paths
| representing a sub project. That's what josh is trying to
| solve: a "virtual" repo that behaves like a real git repo but
| behind the scenes seemlessly integrates with the big
| monorepo.
| Jenk wrote:
| disclaimer: I haven't seen for myself what benefits
| monorepos actually provide so I don't fully grok them.
|
| This is the kind of talk about monorepos that makes me
| think they are a bad idea. Why would someone want to
| maintain a monorepo and then pretend it's not a monorepo?
| Not only just pretend it isn't, but invest not-
| insignificant time on the problem of pretending it's not a
| monorepo?
|
| I am immediately thinking of the horribleness of how some
| of the (older) javascript frameworks re-invented the back
| button (and browser history in general) instead of.. ya
| know, using the browser.
| jayd16 wrote:
| Its simple. Monorepos allows(does not force, nor
| guarantee) you to make bigger atomic changes to many
| projects at once.
|
| You can update a library and all the downstream projects
| in a single commit. There's no race condition or caching
| problem of pulling an update without pulling/seeing the
| dependency update. You don't need to wait for dependency
| artifacts to build and propogate.
|
| You can create a turn key build script that will build
| the world from source. You can skip any local artifact
| storage like Artifactory. You don't need to pull multiple
| repos in a serial fashion, no dependent pulls. You can
| structure your codebase such that if you pull one commit
| it can have no other dependencies.
|
| The draw back is Git happens to not make it easy to pull
| just one folder. Other things like Perforce make it
| trivial.
| gumby wrote:
| One big advantage of a monorepo is that when you check
| out the tree you automatically get the versions of all
| the files that work together (assuming there's some CI!).
| If you want to refactor an API you can refactor its
| callers easily and check the whole thing in. Etc.
|
| With each project having its own repo, then you have to
| track the fact that Foobar 2.2 works with baizo 1.6-1.8
| but not more recent versions.
|
| Also conceptually it's easier when you are working with
| the client and the server at the same time, or the two
| mobile apps, and so on.
|
| Of course people manage without this when the project has
| stuff that doesn't fit in a software repo (CAD designs,
| artwork, etc...there's a reason why that POS Perforce
| survives, for example). Solidworks has its own
| proprietary RCS that doesn't work with anything else.
|
| IMHO if the project is relatively small (say <500K LoC) a
| monorepo is almost always the way to go. But with a big
| project it breaks down.
| vlovich123 wrote:
| That's an interesting claim considering Google, Facebook
| and Microsoft run monorepos. Heck Apple does too
| internally although just for the build team (snapshots of
| each project submitted to them, but it all goes into a
| mono repo)
| gumby wrote:
| Google's monorepo isn't complete, excluding various
| things like Android. Also it isn't git
| romwell wrote:
| I think the parent comment meant that multi-repo breaks
| down for large project, but made a typo.
| romwell wrote:
| >IMHO if the project is relatively small (say <500K LoC)
| a monorepo is almost always the way to go
|
| A typo? You seem to mean that multi-repo is the way to go
| :)
| beagle3 wrote:
| It's basic git that breaks down, not the monorepo model.
|
| And it's less about LoC, and more about the number of
| files and how much binary stuff you put in your repo (and
| how often it changes). Git is really bad when binary data
| is involved.
|
| Git has the facilities to keep monorepos clicking along
| (shallow clones and sparse checkout) but they aren't
| along the "happy path"
| romwell wrote:
| That's why there's git-lfs[1] (large file storage)
|
| Keeps the binaries out of your repo, replacing them with
| pointers
|
| [1]https://git-lfs.github.com
| drjasonharrison wrote:
| And many git repository hosting services, like bitbucket,
| have limits on how large your repository can be.
| Sometimes you can upgrade these limits. This often leads
| to fear of exceeding this limit and can lead to one
| repository per module.
| dboreham wrote:
| Monorepo is an alternative to having binary dependencies
| with a registry scheme such as npm or maven (at the
| organization level). It's essentially only workable with
| tooling support none of us has (unless you work for
| Google or one of the few other shops that have said in-
| house tooling). It isn't a workable approach using stock
| git or github (but that won't stop people trying nor
| claiming to the contrary).
| mumblemumble wrote:
| I wouldn't want to do it at Google's scale without
| Google's tooling. But my experience has been that, at the
| scales I've worked at (no more than a couple million LOC
| across the organization), the limiting factor isn't
| source control, it's the build system. Maven, for
| example, doesn't really understand monorepos, so it can
| be a bit difficult to figure out a how to implement a
| policy for deciding what needs to be built when that's
| less heavy-handed than, "build everything always."
| zeven7 wrote:
| > I am immediately thinking of the horribleness of how
| some of the (older) javascript frameworks re-invented the
| back button (and browser history in general) instead of..
| ya know, using the browser.
|
| They did that because the browser didn't support adding
| to the history via JavaScript.
|
| But even now that the browser does support adding to the
| history via Javascript ... is that really just "using the
| browser"? At some level in many modern web apps back
| button history is not just the browser. This isn't an
| ancient thing left behind with old frameworks.
| beaconstudios wrote:
| When you have shared resources between two services (such
| as React components), you have 2 choices: have a separate
| shared repo that will need to have its own versioning,
| and keeping it in sync with development is a pain if
| multiple people are working on features that touch both
| shared components and individual services at once, or
| have a monorepo where the service and shared components
| can just be worked on through the same repo.
|
| The same story is true with things like APIs or types
| where two services need to stay in sync.
| mumblemumble wrote:
| Whereas what I was kind of hoping for was something that
| works like svn externals.
|
| (No, git submodules are not it.)
| lbhdc wrote:
| I believe git provides that functionality through sparse-
| checkout. You can clone a repository without checking it
| out, then use sparse-checkout to only pull the paths you
| want.
| rcxdude wrote:
| sparse-checkout only reduces the number of files copied
| from the local repo to the working directory. It doesn't
| affect the amount downloaded data. For that you need
| shallow and partial clones (shallow clones give you a
| subset of history, partial clones give you a subset of
| the files within that history). Partial clones especially
| are a relatively new and not heavily used git feature.
| geitir wrote:
| Google's tool repo kind of does what you thought this does
| icythere wrote:
| Is that for android build system only? West is a better one I
| think https://github.com/zephyrproject-rtos/west#basic-usage
| jazzkingrt wrote:
| I think this is quite exciting, as it solves a major unsolved
| problem for large git monorepos: enabling development or CI/CD
| inside a git monorepo without requiring a large checkout.
|
| As monorepos grow huge, this comes to be very costly or even
| prohibitive, and companies like Google simply don't use Git.
|
| Here are some problems with alternative approaches that have been
| mentioned:
|
| * VFS for Git: I believe abandonded by MSFT in favor of improved
| client-side tooling:
| https://github.com/microsoft/VFSForGit/blob/master/docs/faq.... .
|
| * Sparse checkout: limits ability to use a build system to
| dynamically find any dependencies and rebuild them
|
| * Submodules: can't atomically update both the parent and the
| child repo, have to manually update the referenced commit of the
| child repo in the parent repo, and each collaborator must
| manually update their child repo when the commit changes
| bastardoperator wrote:
| VFS is being replaced in favor of
| https://github.com/microsoft/scalar
| WorldMaker wrote:
| Which scalar is "mostly" just a config tool for git sparse
| checkout of git partial clones with git commit-graph support
| turned on. All of that is stuff contributed directly into the
| git client.
|
| Beyond that "mostly", it also configures git lfs, which
| likely will always be a git plugin and not directly in the
| client and the rest of it seems like stuff Microsoft is
| testing before upstreaming it directly into the git client.
| maratc wrote:
| > CI/CD inside a git monorepo without requiring a large
| checkout.
|
| this can also be solved by using a git mirror.
| jazzkingrt wrote:
| Doesn't this just provide another (perhaps more nearby)
| remote?
|
| You still end up doing some kind of large checkout.
| JohnHaugeland wrote:
| I have yet to hear anyone give me a coherent explanation of why a
| monorepo is better.
| DethNinja wrote:
| This can be kinda achieved with git submodules.
|
| I'm currently using a monorepo with submodules and it works
| really well.
|
| Dependency management is not a huge issue too, at least if you
| don't have hundreds of submodules.
| tazjin wrote:
| It can not - josh can do many arbitrary kinds of (reversible)
| transformations on the repo, which allows you to have different
| external "projections" of your monorepo.
|
| Imagine a company that develops strongly interdependent
| software in a monorepo, but needs to publish different subsets
| of this software to external entities which also expect a
| coherent version history.
| mugsie wrote:
| git filter-branch would fit this use case?
| chrschilling wrote:
| On a basic level, yes, both Josh and git filter-branch do
| essentially the same thing. The difference being that Josh
| is _much_ faster not just compared to git filter-branch but
| also compared to all the other similar tools out there,
| especially when run repeatedly in the same repo.
|
| Also being a server it does not require any installation or
| resources on the developers machine.
|
| In addition to that over time more features where added
| that git-filter branch does not have, most notably "josh
| workspaces" which is a DSL for repo transformations.
| zactato wrote:
| I might be reading this wrong, but it seems like this creates a
| second place where a project would need to express dependencies
| between modules.
|
| You would need to do it for JOSH and whatever your build tool is.
| You'd probably need some additional git commit hooks to ensure
| your build tool of choice config is in sync with josh.
| icythere wrote:
| There are a few (re)solutions in the wind. The latest one that
| I've known is `west` (part of Zephyr-RTOS project), but I haven't
| tried yet.
|
| There may have wrong description (FIXME) but a sort list is found
| here: https://github.com/icy/git_xy#why
| nemetroid wrote:
| How does this compare to partial clones and sparse-checkout? This
| question was raised as an issue in the project but was closed as
| "not really an issue", which I guess is true.
|
| https://github.com/esrlabs/josh/issues/23
| villasv wrote:
| Technically correct, but so unhelpful. No way I'm using a
| project that has this kind of community engagement.
| jakevn wrote:
| I wouldn't even agree that it's not an issue. It's something
| the README doesn't cover.
| sjburt wrote:
| Generally, Github Issues are used as a bug tracker, not a
| community FAQ. Asking a maintainer to compare and contrast
| their project with another project or git feature seems a bit
| demanding.
| villasv wrote:
| Keyword: generally. Plenty of projects do allow community
| questions, specially small ones or early stages. Is there
| anywhere else to ask that question? If there is, it isn't
| prominently signified, answering at least "ask this >here<"
| would be common sense. At a minimum, this issue evidences
| the need of documentation and should be addressed in some
| way with more than "I don't have to answer this".
| bjtitus wrote:
| GitHub Discussions is a much better place to handle Q&A:
| https://docs.github.com/en/discussions
| detaro wrote:
| Which didn't exist when that issue was created.
| wildmanxx wrote:
| And none of the people in this thread did provide even a
| hint of an answer either...
| detaro wrote:
| It's not a trivial question, so it's not that surprising
| that nobody answers it without prior knowledge of the
| thing?
| derefr wrote:
| IANAMU (I Am Not A Monorepo User), but as far as I understand
| monorepos and their advantages, I'm not sure what the use-case
| for this tooling is.
|
| Most of the time, when an org chooses to move to having a
| monorepo (rather than just being left with one by accident of
| history), the key advantage they're striving to attain, is the
| ability to make changes to cross-cutting concerns across many
| distinct applications/libraries, with single commits/PRs. To
| change an API, and _all_ of its internal callers, atomically,
| without having to worry about symbolically binding the two
| together with dependency version constraint resolution.
|
| Which is to say, the key advantage of a monorepo comes from
| having the whole monorepo checked out locally.
| oftenwrong wrote:
| I think there are some potential applications.
|
| For example, you want to have a project in the monorepo, but
| also share it outside of your organisation, and even have
| external contributions brought back in.
|
| Or, you are working in a project in the monorepo, but you don't
| want to (or can't) check out the entire repo. You can still
| checkout just that project and its deps.
|
| Or, you are working in a polyrepo-using organisation, and you
| want to experiment with using a monorepo while continuing to
| allows devs to work in the polyrepos that it is composed from.
| No big-bang cutover necessary.
| chrschilling wrote:
| Yes that it exactly the idea. Translating between both mono
| and poly repo, partial sharing with others (distributed
| development) and gradual adoption without a big bang.
| majormajor wrote:
| When I've used a monorepo, that was one of the explicit goals.
|
| Avoiding "here's my new library version, go see if it breaks
| your shit" was the goal - you make a change, you run the tests,
| you see if the whole company's code can still build or not.
| Having fully-separate projects in directories in the monorepo
| using published dependencies was considered an antipattern
| (though it was very hard to keep some teams from doing that).
|
| The disadvantages of the resulting monorepo weren't "this
| directories are so big to keep checked out when I'm just
| working on one specific project" it was "our old build times
| and build tools are dying under the strain and even trying to
| move to a 'monorepo friendly' build tool might be an
| intractable problem because our dependency graph has become
| such a mess of spaghetti."
|
| A monorepo that was done well from the start so you _don 't_
| have the slow-spaghetti-build problem from months or years of
| "oh it's easy to depend directly on this full other module,
| let's just do that" sounds very appealing. We just didn't pull
| it off in practice, and this project would ... maybe... help in
| the early stages by letting people have more restricted
| checkouts? But only if you already know what you're doing
| anyway.
| roughly wrote:
| The basic problem with trying to solve the problems with
| versioned releases with monorepos is that the code in the
| repo and the code in production are not the same. If you
| change your service's API, just updating all of the other
| clients of your service in the repo is not sufficient to
| skirt the problem of different clients expecting different
| APIs.
|
| Re the tooling issue, I've frankly never seen it solved. I
| presume Google's done it, because they're a bit messianic
| about the whole thing, but outside of them, I suspect all
| monorepo implementations fit into the "Communism" model of
| working great in theory.
| spicybright wrote:
| I don't understand the problem. Code structure that listens
| to dev/prod mode can handle the different APIs. Branches to
| mark prod vs. dev.
|
| Can you give a more concrete example of the problem? I
| don't think I'm understanding.
| thebean11 wrote:
| The issue comes if you are communicating between
| different binaries (for example, RPC requests or message
| handlers). Just because you can change the code
| atomically doesn't mean you can deploy the client and
| server atomically.
|
| As someone pointed out though, proto mitigates a lot of
| this.
| roughly wrote:
| The pitch I often hear with monorepos is "When you change
| your service/library/whatever, you can simultaneously
| change all of the clients of that
| library/service/whatever in one PR". The problem is a PR
| is not a release - just because the code in the repo is
| up to date doesn't mean the code in production is. So,
| for example, if your service changes to require an extra
| parameter in the API, sure, you can update all of the
| clients of that API to provide that extra parameter, but
| that still requires them to be built & released as well -
| just updating the code doesn't actually fix the
| deploy/release coordination problem.
|
| Short version, you're right, coordinating releases with
| feature flags, branches, etc. solves this problem, but it
| solves the problem in a multi-repo world too, and a
| monorepo doesn't obviate the need for those solutions as
| many of its proponents seem to suggest.
|
| There are probably legitimate uses for monorepos, but an
| awful lot of people seem to position them as a silver
| bullet that lets developers stop having to worry about
| coordination in a multi-service world, and that's just
| not how it works out.
| floober wrote:
| I agree that it doesn't improve things at the service
| interface level. What I have found is that I often have
| several internal libraries, and a monorepo ensures that I
| know if a change to my library breaks another service
| because they're all in one build. Keeping things
| consistent across services is very useful in my
| experience.
| throwaways885 wrote:
| > If you change your service's API, just updating all of
| the other clients of your service in the repo is not
| sufficient to skirt the problem of different clients
| expecting different APIs.
|
| Most Google APIs are written in protobuf, which is
| backwards compatible over the wire. This makes most changes
| painless, although caveat emptor as things can still get
| messy as they do in the real world. With something like a
| Java interface, it doesn't really matter if those APIs
| change because all the code is built and released together
| at HEAD.
| majormajor wrote:
| Backwards compatible from a parsing-this-message-
| won't-blow-up standpoint but not necessarily from a
| semantic meaning-of-each-field standpoint if your devs
| are ... clever ... enough. ;)
|
| That discipline to commit to an API and meaning for each
| field and not keep tinkering with it in messy ways is an
| under-valued and under-evangelized aspect.
| roughly wrote:
| > With something like a Java interface, it doesn't really
| matter if those APIs change because all the code is built
| and released together at HEAD.
|
| Right, this is the thing that winds up following from the
| monorepo - the monorepo problem goes away if we just
| release everything, the whole fuckin' system, into
| production every time we make any update to master. If
| that's your system, hats off to you, I'm not nearly that
| good of an engineer.
| barrkel wrote:
| It's actually easier, in many ways -- if you can manage
| to release continuously. It forces you to stage your code
| incrementally and you should be able to roll back things
| easily if you see breakage.
| [deleted]
| qznc wrote:
| You seem to mix two different goals here. Atomic commits
| across everything surely is a feature of a monorepo which a
| manyrepo setup cannot replicate. However, the "test whole
| company's code" could also be achieve with a central package
| manager and many repos. That is what Amazon does:
| http://beza1e1.tuxen.de/amazon_manyrepo_builds.html
| chrschilling wrote:
| IATAOJ (I Am The Author Of Josh) ;)
|
| You are absolutely right about the main motivation of using a
| monorepo: Allowing upsrteam library maintainers to see
| downstream usage of their code and make the required downstream
| changes themselves at the same time they change their
| libraries.
|
| Also like you say the easiest way to get those advantages is to
| just check out the monorepo locally, so if there are no other
| reasons preventing you from doing just that, go for it.
|
| However there are a few reasons why this is not always
| sufficient:
|
| Size: The repo might be so large that cloning it all will makes
| local tools (git cli, guis,...) slow to use, or in the most
| extreme case require to much disk space for your machine. To
| address this there are some git native tools like partial clone
| and sparse checkout, so size alone is not really the the main
| issue for us.
|
| History "pollution": Having a lot of somewhat loosely related
| projects in one tree means a history that shows all the
| changes. Yes git can filter them, but once again that _might_
| be a performance concern, but once again not really the biggest
| motivation to create a new approach /tool.
|
| Permissions: In some organisations (like the one I work for) it
| is not possible to give all developers access to all the code
| and thus the advantages of monorepo get lost just by trying to
| comply with data protection standards. The only solution with
| native git is to split the repo at legal (not necessary
| technical) boundaries and try to coordinate the changes across
| those. Loosing most of the benefits described. Josh does not
| have a full blown permissions system yet, but the concept
| certainly allows for it and implementation is work in progress.
|
| Sharing with others (aka, distributed VCS): This is the biggest
| motivation for using something like Josh. The partial repos are
| repos in their own right and all the distributed features of
| git can be used with them. In a monorepo setup as you describe
| distributed workflow is sacrificed for monorepo advantages.
| Only developers in the same monorepo see the same sha1s and can
| easily exchange changes. In Josh the same library can be part
| of different monorepos at different organisations and while the
| monorepos have different history and therefore sha1s, the
| "projected" or "partial" library subrepos will have compatible
| history with identical sha1s. In this way Josh can serve as a
| bridge between organisations using different repo structures.
| thrtythreeforty wrote:
| This is a really interesting approach. How would you layer CI
| logic on top of this? Given your example workspace josh file,
| dependencies = :/modules:[ ::tools/
| ::library1/ ]
|
| how are the canonical build artifacts for, say, ::library1/
| determined, and how are they presented to the workspace?
|
| I understand that the partial repo layering is the key
| innovation that exists a layer below what I'm talking about,
| but I'm trying to understand how you can ergonomically layer
| never-build-twice logic on top of it.
| wildmanxx wrote:
| The ability to do _some_ changes like that is not the same as
| doing it _always_. Most commits are quite localized, and those
| should not be penalized by the ability to have a few cross-
| cutting ones.
| bognition wrote:
| They are very useful when you've got hundreds of engineers
| working on distinct projects that are loosely connected.
| Imagine you have a team working on core libraries, a few
| product platform teams, and finally a team working on a
| customer facing feature. The feature team commits to master and
| a build kicks off (build 1), that build fails for an unrelated
| issue (say the build node dies) and the build kicks off again
| (build 2). With multiple layers of libraries supporting the
| feature team, its entirely possible that a dependency could
| have changed and the end result of build 2 would be different
| than build 1.
|
| When each commit shares a common timeline it is really easy to
| rebuild build 1 with the exact same dependencies.
| derefr wrote:
| Git submodules already solve that problem, though. As does
| publishing all the core libraries etc. as language-ecosystem
| packages on a private package namespace or internal corporate
| package repository, and then resolving/locking the language-
| package dependencies to specific tags/refs with a lockfile
| that gets committed to the downstream repo.
|
| These are the "obvious" solutions to this problem, the first
| ones the average software architect would reach for. What
| would lead them to ignore these options and choose a monorepo
| instead, if not for what I mentioned above -- the ability to
| make atomic changes to cross-cutting concerns?
| chiefalchemist wrote:
| I've resisted - for too long? - submodules as I've always
| heard they can be problematic; that the actual
| implementation isn't quite what I should be.
| justaguy88 wrote:
| Git submodules have an annoying attribute: double commit
|
| First you commit/push to one repository
|
| Then you update the submodule pointer in the parent
| repository
| zerkten wrote:
| >> These are the "obvious" solutions to this problem, the
| first ones the average software architect would reach for.
|
| They would have been, except the experience working with
| them was somewhat problematic for those who have been down
| this path, and the message that many devs have received is
| to avoid them. I kind of agree with going to submodules,
| but confidence doesn't seem to have been rebuilt.
| Glavnokoman wrote:
| Having been to the both sides of it I can say there exist exactly
| 0 advantages of a monorepo setup.
| zaphar wrote:
| I have also been on both sides and I'll say that each side has
| different sets of advantages highly dependent on a host of
| factors a non exhaustive list of which is:
|
| * Culture
|
| * CI/CD tooling support
|
| * Codebase sizes
| geschwindner wrote:
| At our startup, we chose to start with a monorepo. Our team is
| small, but one of the big advantages we've had so far is
| avoiding the n*m (for n services and m tools) problem with dev
| tooling - which leads to a very smooth developer experience.
|
| For example, to run one or more services locally, we use a
| single script that sits at the repo base - 'dev.sh
| service1,service2,...'. This avoids a lot of headaches for our
| developers, as we enforce compliance when adding a project to
| the repo. Lint config? One to rule them all. Test coverage
| thresholds? Single one. This consistency is the biggest win in
| my opinion.
|
| Similarly, our integration tests are very easy to write without
| commit skew.
|
| Finally, sharing libraries has been painless - since we have
| common/ and common/third_party/ directories at the monorepo
| root.
| mdtusz wrote:
| Another benefit (imo one of the biggest ones) is not having
| to constantly create releases across all your repos and
| manage their states when jumping around during development.
|
| My previous job used a collection of about 6 repos for
| different services and such, and it was a constant struggle
| to ensure the correct versions were used in development -
| especially if you were working on a bigger feature that
| wasn't yet released but required "future" versions from other
| repositories.
| Uberphallus wrote:
| Exactly this.
|
| Why would one want to do n pull/merge requests, n separate
| reviews (of related code), and n deliveries for a single
| evolution, is something I can't understand.
| andix wrote:
| Simplicity is always an advantage.
| indymike wrote:
| You can have your simplicity one of two ways:
|
| * Devops - one repo, one deploy, let the developers figure
| out which repo is which.
|
| * Developer - one codebase, let the devops people sort it out
| and write lots of tooling to make my monorepo work.
|
| If this stuff was easy, everyone would be a developer.
| 0xbadcafebee wrote:
| They definitely need a long FAQ. I'm sure it's different than
| submodules _somehow_ , but in what ways/circumstances/purposes, I
| have no idea.
| MakersF wrote:
| Like some people, I was expecting to find a way to have the
| advantages of a monorepo while having projects in separate repos.
| This is something Bloomberg is doing, and it's very cool. Each
| project is a separate repo, but they have a central integrated
| "repo" with all the repos, which is the "source of truth", and
| were code is built and deployed from. You can commit changes in
| your repo, and then you "release" the code into the integrated
| repo, which will rebuild all the transitive dependencies and run
| their tests to make sure everything still works. If anything
| fails, your release of the code is not merged in the repo. I'm
| now working with a monorepo, and I much prefer the Bloomberg
| approach. Cross repo changes can be made atomically (you update
| the reference in the integration repo for multiple individual
| repositories at once), and that is usually the big sell point of
| monorepo. And it doesn't have the downsides of the monorepo. The
| only issue is that it's not very ergonomic, and there isn't a
| tool to make that easy. But building such a tool is definitely
| easier than implementing a virtual FS as it has been done in
| multiple companies.
|
| I'd love if someone still working there were to write a nice post
| about that system, it was the first of such a kind I saw.
| throwaway315724 wrote:
| > I'd love if someone still working there were to write a nice
| post about that system, it was the first of such a kind I saw.
|
| I don't know how far back you saw the Bloomberg system, but at
| this point it's basically the same as the Debian system (as in,
| debian/ subdirectories, .deb files, etc.). Versions of git
| projects are published as tarballs (source packages). Then sets
| of published projects are "promoted" and all projects that
| transitively depend on them are rebuilt and unit tested in a
| sandbox environment. If that process fails, the promotion
| fails.
|
| Each source package can use any number of build systems,
| implementation languages, or project structures.
|
| There's also a legacy subversion monorepo with a monolithic
| build system that builds on top of that, but it's slowly being
| phased out.
|
| All that is an integration build including thousands of
| discrete projects. Those projects typically have additional
| CI/CD enrollments outside of the integration build system too.
| choeger wrote:
| But that only works in one direction, no? So it works if you
| can develop your single repo, but it doesn't help you when you
| depend on other projects.
|
| I think the best approach would be to have bidirectional links
| between the projects (if A needs B, then A has the stable
| version of B and vice versa). The point in that setup would be
| that "upstream" projects can notice when they are about to
| break tests in "downstream" repos and act accordingly.
| MakersF wrote:
| They do, the repositories define dependencies, so when you
| change something everything that depends on you is rebuilt
| and tested. This prevents breaking changes, both for your
| dependencies and your reverse dependencies, identically to a
| monorepo.
|
| It's a bit complicated to explain, but it works. That's why I
| hope they'll make a blog post :)
| klysm wrote:
| A lot of the comments here are surprisingly dismissive. I think
| having the ability to project parts of your git repo (still with
| a normal git api!) is an incredibly useful feature. Take the
| example of DefinitelyTyped: the maintainers can just do all the
| things they want in one repository which _vastly_ reduces the
| development overhead, but consumers of that code can use it
| however they please. If I understand correctly, you could have a
| submodule reference work out of the box but to a subset of that
| repo that you care about - seems pretty damn cool to me!
| chrschilling wrote:
| What you are describing is one of the main use cases at ESR
| Labs (where Josh was created): For developers it is very
| convenient to work in a single tree. For reviewers and CI it is
| useful to look at the changes in a larger context. For
| consumers/integrators however it is useful to only look at
| parts of the code that have to be shipped to particular
| customers, as submodules(or the like) in their repos. Plus a
| lot of package managers assume library == repo as a default, so
| it is also easy to integrate with those while keeping monorepo
| processes for development.
| villasv wrote:
| Before the traditional HN comments "isn't this just ...?", the
| README already does that for you:
|
| > a blazingly-fast, incremental, and reversible implementation of
| git history filtering
| im_down_w_otp wrote:
| I've come to learn that git monorepos are becoming quite popular.
| What I don't understand is why people are using git for this kind
| of workflow. It forces you to actively work against git's design
| goals and implementation. Which then compels the use of several
| odd workarounds and kludges to kind of seemingly reassemble a
| half-baked flavor of subversion. Why not just use a tool designed
| around monorepos and subtrees? I'm genuinely curious. I assume
| I'm missing something.
| an_opabinia wrote:
| > It forces you to actively work against git's design goals and
| implementation.
|
| It works against _GitHub 's_ design goals. You could work on
| multiple, logically distinct projects in the same git
| repository easily, if you so choose. git was designed for
| Linux's workflow.
| ot wrote:
| The problem is that the alternatives are much, much worse.
|
| There's nothing about the git/mercurial object models that
| makes them intrinsically inefficient with monorepos.
|
| What's inefficient is materializing the object database (when
| cloning) and the working copy (when checking out), when you're
| only going to need tiny portions of them.
|
| Subversion doesn't have the first problem (but comes with
| extremely slow history operations), but sparse checkouts don't
| really solve the second because you have to statically know
| what to filter.
|
| A better direction is instead to virtualize the filesystem, so
| you get the semantics of a real monorepo with git/mercurial,
| and you fetch only what's actually needed without any change to
| your tooling (it just needs to interact with the filesystem).
| It's also very easy to transparently implement caching and
| prefetching this way.
|
| This is the approach that Facebook and Microsoft took, and I
| believe Google too.
| WorldMaker wrote:
| Though it is interesting that Microsoft's own efforts have
| been moving away from the virtual filesystem approach and
| back towards making sparse checkouts better/more reliable and
| better/more reliable support for partial checkouts (of
| history especially) and better memoization and caching of
| history reachability information (git commit-graph).
| jayd16 wrote:
| No other DVCS is as popular. Git has a vast ecosystem of tools.
| No one wants to drop that.
|
| Its certainly not against the design goals as official tooling
| supports shallow and sparse checkouts, they're just
| experimental features.
|
| We should just strive forward and continue to make these
| features good.
| endisneigh wrote:
| What are these tools designed for monorepos and sub trees?
| fnord123 wrote:
| perforce helix core
| usrnm wrote:
| Perforce?
| im_down_w_otp wrote:
| Probably the most popular one would be the "new hotness" that
| everyone used, or aspired to use, before git became popular,
| Subversion (https://subversion.apache.org/).
|
| There are others that aren't free (e.g. Perforce) and some
| that aren't quite dedicated to the monorepos & subtree
| workflow, but which handle it better by design (e.g. Darcs,
| http://darcs.net/).
|
| But, mostly I'm thinking of Subversion.
| triceratops wrote:
| Subversion on the server used with git-svn is bearable.
| Most of the advantages of git (local commits, rebasing,
| easy branching and merging) manifest themselves on the
| client-side.
|
| The problem with using svn on the server is the lack of
| good tooling for things like code review. There's no SvnHub
| or SvnLab.
| dalyons wrote:
| haha yeah, noones going to go back to subversion. The sum
| of the pain of the SVN problems solved by git is much
| larger than the slight pain of using git for a mono-repo.
| 0xbadcafebee wrote:
| The reason I would use a Git monorepo over Subversion is
| that Subversion is really painful to use. I would rather
| workarounds and kludges than that tire fire.
| im_down_w_otp wrote:
| That was never my experience with Subversion, but YMMV.
| [deleted]
| gedy wrote:
| A big chunk of the industry joining in past decade has never
| seen/used anything but git, so it's the one hammer they have
| available.
| quotemstr wrote:
| Is that even a bad thing? If people are familiar with git ---
| its command syntax, its commit model, its collaboration setup
| --- why not let them keep using this model? Isn't it better
| for everyone if we put a few hundred people on targeted
| scalability fixes for git instead of making a few million
| people drop productive work and learn a new tool?
|
| I mean, most of the people in industry today have known no
| character encoding except ASCII and its supersets --- and
| that's a good thing!
| Uberphallus wrote:
| Git, while better now, it still allows people who are
| _just_ familiar to shoot themselves in the foot.
| quotemstr wrote:
| So? It's the standard. Anything else starts with -1000
| points. Something like hg might be better, but not better
| enough to be worth the cost of breaking uniformity in the
| industry.
| prionassembly wrote:
| I never get the footgun argument. If you need a gun,
| you're liable to shoot yourself. Programming in general
| is more and more accessible to wider audiences, but it
| hasn't become intrinsically easier.
| SquishyPanda23 wrote:
| > What I don't understand is why people are using git for this
| kind of workflow.
|
| People like to use distributed workflows even with monorepos.
| E.g. chains of commits, branches, rewriting local history, etc.
|
| It's clear that people want a mixture of monorepos with
| distributed workflows. There are two ways to get there: add
| distributed flows to a monorepos, or build a monorepo layer
| over a distributed tool.
|
| Both seem like valid approaches. The market will decide which
| approach it prefers.
| andix wrote:
| That looks nice. It must have some major drawbacks? Sounds too
| good to be true...
| oofbey wrote:
| It's definitely more complex than just using a monorepo. This
| tool that all your code runs through is young and not well
| supported.
| runawaybottle wrote:
| Most of you don't need a monorepo, the same way most of you don't
| need, well, half the shit peddled in the tech Instagram
| (conferences, meetups, mediums, blogs, hn).
|
| You just don't need that stuff, there's like 20 of you on a team
| and at best your app probably sucks and barely has users, and if
| it does have users, it's probably some trivial bullshit.
|
| You're all a bunch of ordinary folks, so stop fucking up the
| workplace with your identity crisis. No, you are not an elite
| engineer, you are Bob, the guy who goes home every day and
| watches Netflix/plays video games.
| sergiomattei wrote:
| Surprisingly, a monorepo is much easier for smaller teams and
| individuals to work with.
|
| As someone who has been maintaining a React and Django app solo
| for the past three years, two repositories or more is too much
| cognitive overhead to work with.
|
| Never doing that again. Monorepo are easier for small apps.
| runawaybottle wrote:
| Putting stuff in the same repo is a pragmatic idea. Going
| into the monorepo isolated self contained publishable
| app/package is a whole 'nother thing, along with all the
| tooling necessary to make it work seamlessly.
|
| You want to put stuff in the same repo, that's fine. What's
| with all the other bullshit?
| detaro wrote:
| The nice thing is, you can replace "monorepo" with "multiple
| repos" in your comment and it's just as believable a statement
| arguing for avoiding trouble with coordination/packaging etc.
| runawaybottle wrote:
| Well, it's something that needs to be said about a lot of
| things. Life is a balance and now days in tech I see the
| pendulum swinging way too far to the other side.
| taeric wrote:
| The problem with trying to force one single history, is that it
| ignores deployments. And user onboarding/behavior change. All of
| which can be relevant when working on a project.
|
| With multi repository projects, this helps some thinking, as it
| is clear that the changes were not atomic between systems. They
| are literally separate at all layers, including the commit.
|
| I sympathize with wanting a simpler view. I'm just worried on an
| inflated value proposition.
| klodolph wrote:
| > The problem with trying to force one single history, is that
| it ignores deployments.
|
| I'm not sure what, exactly, the problem is that you're talking
| about. When you deploy something, it's built from a specific
| commit.
|
| Deployment is not atomic, it's a gradual process that takes
| some amount of time. Between when you (or your automation)
| chooses to deploy a system and when the deployment finishes is
| some window of time. The system will often spend much of that
| time in a partially updated state. You may also choose to
| canary changes, so you will have a mix of different versions in
| production at any given time. At companies where I've worked,
| the time from deployment start to finish for backend systems
| ranges anywhere from hours to weeks.
|
| I don't understand how this relates to multi-repo or mono-repo
| concerns, however. The repo is a history of the source code
| (intentional changes by humans), it's not a history of the
| state of your production systems (which are the results of
| automation).
| taeric wrote:
| Many of the mono repo guides I see are off the "all projects
| in one repository" kind. If the project doesn't build and
| deploy as an atomic unit, then having atomic code changes is
| a foot gun.
|
| It is amazing how many times I've seen folks think that just
| because they can get build time tests happy with changes in
| two projects, that they can safely send out the two changes.
|
| Does a multi repo "solve" this? Of course not. But it is
| easier to reason that two projects clearly need two deploys.
| Versus having to remember that one commit could be N project
| deployments.
| klodolph wrote:
| > ...then having atomic code changes is a foot gun.
|
| Why is it a foot gun? I don't know what the negative
| consequences are here.
|
| Atomic commits are just to make development easier. It
| means that you can refactor downstream dependencies in the
| same commit that you make a change to an upstream library.
| This way, you either build and deploy the new library + the
| refactored downstream dependency, or you build and deploy
| the old version, but never some mix. It reduces the number
| of possible configurations that can be built & deployed,
| since you can only pick from a point in one repo's history,
| and you can't mix and match various points in the history
| of different repos. With multi-repo, it is harder to
| discover down-stream dependencies.
|
| > It is amazing how many times I've seen folks think that
| just because they can get build time tests happy with
| changes in two projects, that they can safely send out the
| two changes.
|
| Build-time tests don't catch 100% of errors. Some errors
| will still make it into production. I don't see how this
| problem is related to the multi-repo vs mono-repo problem
| _at all._ At places where I 've worked, if you change
| project X and project Y, and your commits pass the build
| test, you still have to submit the commits in some linear
| order. The pre-commit tests for X will include Y or vice
| versa. X or Y will be rebased or merged on top of the other
| one, and the result will have to go through pre-commit
| tests.
|
| I am just trying to understand what problem multi-repo is
| supposed to be addressing here, and I don't have the
| slightest clue what you're getting at.
| taeric wrote:
| I call it a foot gun because of how many times I've seen
| people shoot themselves in the foot.
|
| My assertion is that the "build and deploy new code with
| updated downstream users" only works in a minority of
| cases. Now, I grant that this could be due to the micro
| service nature of where I'm at. And I also grant that it
| is nice when this can work. However, the times it goes
| wrong are usually not at all worth the risk.
| klodolph wrote:
| > I call it a foot gun because of how many times I've
| seen people shoot themselves in the foot.
|
| I understand what "foot gun" means. Explaining what "foot
| gun" means is not helpful. What I don't understand is the
| problem that "shooting yourself in the foot" is a
| metaphor for.
|
| > My assertion is that the "build and deploy new code
| with updated downstream users" only works in a minority
| of cases.
|
| There are two main cases here: libraries and services.
|
| Libraries can be atomically updated no problem, most of
| the time. You change a function and fix the call sites.
|
| For services, you have to add to the API, deprecate the
| old thing, refactor the clients, and then remove the old
| thing after all the clients have been rebuilt and
| redeployed. At least two steps.
|
| What I don't understand is how this would be different
| for multirepo or monorepo setups. In either case,
| removing some old piece of functionality requires waiting
| until the clients have been redeployed. Using new
| functionality requires waiting until the server has been
| redeployed.
|
| > Now, I grant that this could be due to the micro
| service nature of where I'm at.
|
| The teams where I've used monorepos are also the teams
| with the most buy-in to microservice architecture. One
| team I was on ran a service that consisted of something
| like twenty different microservices. These interacted
| with services run by other teams. Everything was in the
| same monorepo. It worked fairly smoothly, as I recall--we
| spent most of our time solving domain problems and
| working on our team's core mission, and I don't remember
| any problems arising from the monorepo setup.
|
| It sounds like your experience is different, and I was
| hoping that you would share some of that experience.
| taeric wrote:
| I have seen people review and ship code that was all
| consistent in a single commit, that had to be deployed in
| separate deployments. In the mixed fleet scenario that
| this leads to, I have seen failures that require
| gymnastics to fix.
|
| Yes, you can do similar with reviews that span multiple
| projects. However, I have seen it happen far more times
| in single repositories than I have in multiple ones.
|
| Again, I am not in any way offering a panacea. I'm just
| saying that seeing things as atomic at one level leads
| people to think they are atomic at the next level. And
| this is a mistake I've seen many many times.
|
| Yes, I have seen people manage it somewhat well. But
| every effort I have been involved with that tried to
| merge code history between projects has had more faults
| of this kind than the other projects I have been on.
| klodolph wrote:
| Ok, that's a good example. I've just never seen that
| stuff make it past code review. Teams I've been on,
| development & deployment are either done by the same
| team, or they're done by two teams who sit next to each
| other. Reviewers are also very skeptical of larger
| changes.
|
| I still don't see advantages to multi-repo here. Even
| within a single project, or single service, deployment is
| often not atomic. If you make a change to service X and
| deploy it, you end up with minimum two versions of
| service X in deployment until the deployment finishes
| rolling out.
|
| So we have automated tests that cover that case... each
| service has to work correctly when combined with not only
| the repo HEAD, but also with the currently deployed
| versions (perhaps more than one! in the case I'm thinking
| of, it was only ever the two previous versions, but other
| teams had longer horizons) Testing against multiple
| versions is a bit more expensive, so it's only done as a
| pre-deployment check, rather than a pre-commit check. You
| cut a branch for deployment, and when one commit doesn't
| play nicely with previous versions of the service, you
| cherry-pick a commit to revert the faulty commit, and run
| the tests again.
| taeric wrote:
| Same situation here. And we have always had tests, too.
| Hard to force mixed fleet behavior in most tooling,
| though. And teams grow, such that relying on code review
| to catch things is really a best intentions.
|
| Putting it in your face that you are changing two
| projects is about the best I can offer here. I would cede
| that this, too, is mainly a best intentions. But having
| as close to 1:1 between code changes, deployment changes,
| and code reviews at least puts things on mainly equal
| footing.
|
| That is, when the explanation of why some code can't
| deploy together is that they are in two projects, and you
| can see that by them being separate repositories, that
| feels easier than knowing that two parts of a single
| repository have to deploy separately.
| klodolph wrote:
| It just seems like such a miniscule benefit, and you have
| to deal with the massive headache of a multi-repo setup.
|
| If you don't want people making changes to multiple
| projects at the same time, it would be trivial to add a
| pre-submit check to enforce that anyway. There's no need
| to switch your entire repository layout just to remind
| people that the code in different folders belongs to
| different projects.
| taeric wrote:
| I will fully cede questionable ROI on this whole
| exercise. But, I will argue that cuts both directions.
|
| That is, I would not push to move from one form to the
| other. I do like multiple projects being as independent
| as I can make them, though.
___________________________________________________________________
(page generated 2021-07-15 23:00 UTC)