[HN Gopher] Cleaning house in Nx monorepo, how i removed unused ...
___________________________________________________________________
Cleaning house in Nx monorepo, how i removed unused deps safely
Author : birdculture
Score : 87 points
Date : 2025-09-29 01:12 UTC (21 hours ago)
(HTM) web link (johnjames.blog)
(TXT) w3m dump (johnjames.blog)
| juujian wrote:
| Not supposed to be a judgemental question. Not every repo has to
| be a labor of love for sure. But how do you get to that place
| where there are 120 unused dependencies? I'm sure there are many
| different pathways where things get a bit out of control. People
| doing the equivalent of pip freeze? Or (too) many cooks?
| rtpg wrote:
| I have witnessed a lot of kinda spurious pinning going on. Or
| like "ok we need to fix the bounds for this transitive
| dependency for a bit" and then it just sticks around.
|
| Over a decade that's once a month, which is a lot though!
|
| I think sometimes people will hear advice like "pin your deps"
| and do a `pip freeze | requirements.lock.txt`, without really
| absorbing that pinning transitive dependencies like this is
| generally not what you want.
|
| You want a lock file! But you tend to want transitive
| dependencies that aren't locked down to get upgraded when you
| upgrade your direct dependencies. But it's a subtlety that can
| get lost in the noise.
| zdragnar wrote:
| 1- I want my dependencies to define a range for the libraries
| that they work with, so we don't have 20 different versions
| of some common library because our dependencies are hyper
| specific
|
| 2- I want every install of my project- be it on a dev machine
| or deploy machine- to have the exact same versions of
| dependencies, including transitive ones. I don't want to deal
| with bugs caused by surprise version changes
|
| 3- if I upgrade a dependency or remove it, I want the
| transitive dependencies managed automatically. I don't want
| orphaned transitive dependencies. In fact, I don't even want
| to think about them at all other than know that they work and
| aren't adding bloat or security risks.
|
| You need a package manager with more than a passing thought
| for handling lock files. For the longest time, npm wasn't it.
| I'd argue that it still isn't, because "npm install" should
| NOT be the command used for both "set up a project for the
| first time" and "add a new package". In the first case, I
| want a reproducible, deterministic result of a known state.
| In the second case, I want to modify the dependency graph,
| producing a new state.
| montroser wrote:
| > I want a reproducible, deterministic result of a known
| state
|
| Yes, that's `npm ci`.
|
| With a fresh `npm install` you get the latest packages that
| satisfy dependencies in the ranges spec'd in package.json.
| That's basically a crap shoot because even if you
| specifically pin the versions of all your dependencies,
| chances are that at least some of your dependencies did not
| have the good sense to do the same for _their_
| dependencies. This is the path to hell.
|
| With a fresh `npm ci` on the other hand, you get back
| exactly to the known state specified in package-lock.json,
| where everything is pinned, all the way down. This is the
| path to happiness.
| rectang wrote:
| Does anybody disable `npm install`? For yourself? For a
| team?
| zdragnar wrote:
| The official tutorial at
| https://nodejs.org/en/learn/getting-started/an-
| introduction-... says to use `npm install` and makes no
| mention of `npm ci` at all. Further, the name of the
| command (though not the clean-install alias) shows that
| it was tacked on top of a fundamentally broken base.
|
| > That's basically a crap shoot because even if you
| specifically pin the versions of all your dependencies,
| chances are that at least some of your dependencies did
| not have the good sense to do the same for their
| dependencies.
|
| As I mentioned in my first comment, I don't really want
| my dependencies to pin their dependencies. I want them to
| specify a range they work with to minimize the number of
| redundant copies of common transitive dependencies.
| chrisweekly wrote:
| PSA: pnpm is fundamentally superior to npm or yarn; used
| properly, it provides reproducible / deterministic builds,
| with efficient control of the dependency graph.
| chuckadams wrote:
| > used properly
|
| That phrase is frequently way more load-bearing than it
| should be. Is pnpm's _default_ behavior the correct one?
| (yarn user here, but open to switching)
| chrisweekly wrote:
| Yes.
|
| No tool can prevent all footguns, but standard /
| idiomatic / out-of-the-box pnpm usage beats even expert
| use of npm or yarn. It's different, and better.
| ziml77 wrote:
| I assure you that hearing "pin you dependencies" is not why
| people create a requirements file using pip freeze. It's
| simply because that has long been how people have said to
| generate a requirements file, because Python spent much of
| its life lacking proper project dependency management.
|
| And now it's extremely hard to get people to stop. There's so
| much info out there on the internet that says to use pip
| freeze that people are going to continue to run into and
| continue to learn to use.
| rtpg wrote:
| my usage of pip freeze has been of the "make sure to know
| what CI used when it build the image" (used to be "make
| sure the installation on the server uses the same packages
| that we used in CI" pre-docker-image world).
|
| Glad that we have better tooling nowadays!
| ipaddr wrote:
| The ecosystem is setup to encourage this type of pollution.
| twodave wrote:
| In my experience it's usually engineers who run install
| commands until it builds locally, then check in whatever
| they've got.
| chamomeal wrote:
| I cannot imagine the work environment where that's acceptable
| lmao
| scorpioxy wrote:
| Unfortunately, that's been the reality in most places where
| I was engaged to maintain a platform. Usually in-
| experienced developers do this and it just sticks and the
| list keeps growing and tech debt increasing until the whole
| thing is no longer maintainable or even operational in some
| cases.
|
| There's never enough time to prune the software just like
| there's never enough time to consider the long term
| consequences of any decision. The incentive system is
| usually set up where the focus is always on the next
| quarter and the short term. And the result is exactly what
| you'd expect it to be.
| wging wrote:
| It sounds like they are counting transitive dependencies. If
| so, that means they deleted far fewer than 120 different lines
| in their own config to end up with that level of reduction.
| thephyber wrote:
| There are several methods you can end up with extra/redundant
| NPM libraries.
|
| At my last company, we had a cleanup project which ended up
| with a similar number of libraries removed over the epoch which
| cleaned up+upgraded NPM dependencies.
|
| (1) the number might be multiplied if one library is removed
| from each package of a monorepo. Eg. If there are 10 packages
| in the monrepo, then you only need to remove on average 12 NPM
| libraries.
|
| (2) sometimes updating the upstream libraries to newer versions
| reduces the size of the upstream dependency tree. Similarly, it
| was more common for NPM libraries to declare a dependency,
| whereas more mature libraries might move those to
| devDependencies or peer dependencies.
|
| (3) removing unnecessary / unused code can unblock you so you
| can remove a library which is no longer referenced in your
| repo.
|
| (4) sometimes programmers over engineer early. Sometimes your
| product requirements change. Reviewing your codebase after each
| of these conditions allows you to decide if you still need all
| of the code/dependencies.
|
| (5) as the NodeJS runtime changes, sometimes you can migrate
| from a 3rd party library to use the built-in tool (we did this
| with BluebirdJS / Promises).
| hinkley wrote:
| Older versions of node especially had problems resolving
| mutual secondary dependencies efficiently, though all
| versions struggle to an extent. Sometimes you have to declare
| a dependency at the top level in order to force the most
| common one to be installed at the root, so the less common
| one gets installed in all of the places it is needed rather
| than sharing the single one at the root.
|
| The likelihood you ever get back to zero copies of that
| module are fairly low, unless it has become persona non grata
| for some reason (leftpad). So it's easy to forget to back out
| the disambiguation entry at the top level. Especially if git
| blame doesn't make it dead apparent (see also people manually
| adding entries in non-alphabetic order so that the next 'npm
| install --save' rearranges five lines).
| bapak wrote:
| > Or (too) many cooks?
|
| Just incompetent cooks. In one of my jobs, I had to cleanup a
| bunch of dependencies like "npm" and "install" because the guy
| typed "npm install" and then pasted "npm install react" or
| something like that. And that was not noticed for several
| months.
| jxf wrote:
| This is a good reason why even cursory checking for whether
| dependencies actually get used is valuable.
| liampulles wrote:
| It happens when devs and businesses reach a point of tolerant
| negligence w.r.t their codebases. I suspect this is true for
| the vast majority of codebases, a "labor of love" codebase in
| the enterprise is a unicorn.
|
| Remember not every project is for some amazing product, with a
| great agile way of working. Some projects are a withdrawals
| system internal to an insurance company, for example. No one
| gives a shit there.
|
| Motivating for any kind of tech debt work on such a project is
| unlikely to be met with approval, and all the devs who really
| care about such things will likely have left the project long
| ago.
| johnjames4214 wrote:
| in a decade+ old fintech monorepo w/ microfrontends + BFFs +
| mixed stacks/design systems (react, angular, express, apollo,
| chakra, tw) the cruft builds up fast.
|
| every time we migrate stuff (like the recent shift to next.js),
| old apps/libs don't always get cleaned out. after years and
| years it's an npm landfill. knip was basically the bulldozer.
| saghm wrote:
| I could easily imagine someone introducing a dependency because
| it's needed for something, then a few months later someone
| refactoring to remove the code that used it but not being aware
| that the dependency wasn't used anywhere else. Repeat this
| pattern a bunch, and the number of unused dependencies will
| start climbing. Keep in mind that the more dependencies you
| have, the less obvious one specific unused dependency in the
| config files will be (in addition to it likely being easier to
| introduce yet another one; it's a lot harder to object to
| dependency N + 1 when you didn't object to the N beforehand).
|
| Without changing the policy around dependencies themselves for
| a project (which presumably would be an uphill battle for
| anyone feeling strongly enough about this), the only way to
| avoid this scenario from happening without tooling would be for
| people to manually verify that a dependency is still used any
| time they remove a usage of it. Asking the person who
| introduces the dependency to keep track of every time someone
| else introduces another usage of it to the codebase would be
| burdensome even if you could assume that they wouldn't ever
| switch jobs/stop contributing to the open source project/take
| vacations when code is merged in their absence, and
| realistically I doubt that asking people to notice whenever
| they remove a use of a dependency and manually verify that the
| dependency is still used elsewhere would be particularly
| effective. I don't think there's any way to make this obvious
| in the absence of tooling, which unfortunately isn't super
| common in my experience even in projects that make ample use of
| linting other static checks in CI.
|
| I don't totally agree with the takes that this is from devs not
| caring about their projects or otherwise phoning it in. Often
| the projects I've seen struggle with things like this (and even
| things that are a lot more worrisome than this, like missing or
| flat-out wrong documentation around stuff like testing locally
| due to the team relying on unwritten shared knowledge of
| process) have developers who care deeply about what they're
| doing, but are stressed, burned-out, and desperately trying to
| tread water to avoid drowning in the sea of things they'd like
| to do to improve things. That's not to say that every situation
| like this is blameless, but unless my experience is greatly out
| of the ordinary, I'd think that quite a lot of us have been in
| or at least seen situations like this enough that we should
| have enough empathy not to jump to conclusions about the state
| of a project being a direct reflection of how much the devs
| care. (To be clear, I don't think this is what the parent
| comment I'm replying to is saying, but it does seem like
| overall there's a bit of a sentiment similar to this in the
| thread as a whole, so it feels worth calling out).
| no_wizard wrote:
| We have a vendors package in our monorepo to centralize both
| tracking and versioning of our production dependencies on the
| frontend.
|
| We pre build the dependencies and upload them to a CDN as well,
| which we can then at build replace the import URLs with the
| stable CDN ones, it works really well. We have thin wrappers
| around each dependency to give a stable interface as well.
|
| For what backend TypeScript projects we do have we created
| typed wrappers around the server framework instead of pre-
| building because those both deploy differently and have
| different constraints but this still lets us track and upgrade
| versions centrally while keeping APIs stable.
|
| Using this approach has had a positive impact on performance
| across the board as we naturally strive to keep our dependency
| count low because the maintenance burden is more obvious
| gdulli wrote:
| The constantly renamed titles and general witch hunt against
| clickbait is really annoying.
| philipwhiuk wrote:
| Because you re-read the same article? It looks like the title
| here is the same as the URL indicating it hasn't changed.
| aitchnyu wrote:
| Can this also detect low-hanging fruit like say, 5 instances
| where we can add some code and give up Lodash?
| johnjames4214 wrote:
| yeah you can definitely point it at lodash (or any utility lib)
| and it'll surface spots where it's imported but never really
| used. but for "we could just rewrite this tiny helper
| ourselves" type stuff, knip won't judge code quality. it's more
| about flagging dead deps/files than saying "ditch lodash." that
| call still needs a human.
___________________________________________________________________
(page generated 2025-09-29 23:01 UTC)