[HN Gopher] Honey, I shrunk the NPM package
___________________________________________________________________
Honey, I shrunk the NPM package
Author : vinnyglennon
Score : 102 points
Date : 2023-10-03 16:55 UTC (6 hours ago)
(HTM) web link (jamiemagee.co.uk)
(TXT) w3m dump (jamiemagee.co.uk)
| hinkley wrote:
| I thought this may be going another way.
|
| I make a periodic practice of searching our node_modules folder
| for files that shouldn't be there and reporting bugs against the
| offending projects.
|
| Usually that's been pretty effective, and now the total cruft is
| around a megabyte whereas before it was somewhere north of 50MB
| all told. (Conditions apply).
|
| coverage reports, test results, build detritus, etc.
|
| The one I'm still debating, because it's becoming a serious
| problem for a couple of our libraries: should the tests be
| included in .npmignore or kept along with the library? I'm not
| sure what the right answer is there. Test sizes especially
| including fixtures can creep up quite a lot over time. I know
| what I'd like it to be, but I'm not sure I can win that argument
| with a bunch of maintainers on different projects.
| diggan wrote:
| > should the tests be included in .npmignore or kept along with
| the library?
|
| What are the reasons it would be a good idea to include the
| tests with the release/distribution of a package?
|
| Seems like something you don't care about when you're just
| using it as a library, unless you want to modify something in
| it, but then you'll clone the library straight from a
| repository anyways, which includes the tests.
| mwilliamson wrote:
| For packages where I don't include tests, I've had at least
| one downstream distro maintainer request that I include
| tests, since at least some of them treat npm or PyPI or
| whatever as the source of releases.
|
| For packages where I do include tests, I've had at least one
| user request that I remove tests so that the footprint of the
| Docker image they're building is smaller.
|
| Both are entirely reasonable requests, but package
| repositories don't really provide a good way of accommodating
| both at the same time, for instance, by allowing a separate
| upload of the dev gubbins such as tests.
| Cthulhu_ wrote:
| The downstream distro maintainer is in the wrong IMO; if
| they want the source code, they can get the source code off
| of e.g. github and roll their own release.
|
| That said, in old Java dependency management (i.e. Maven),
| you could often find a source file and a docs file
| alongside a compiled / binary release, so that you get the
| choice.
|
| But this can also be done with NPM libs already; the
| package.json shipped in the distribution contains metadata,
| including the repository URL, which can be used to get the
| source.
| j1elo wrote:
| As a software maintainer myself, I believe the downstream
| distro maintainer is the one being wrong there.
|
| You have a software project, with a build process, and the
| "output" or final product of that project is the library
| that gets uploaded to NPM.
|
| If they are packaging a software library, they should do it
| from the project's repository, not from one of its output
| artifacts.
|
| They would probably reject a request if someone who was
| downstream of their work decided to repackage their stuff
| and asked them to include tests and other superfluous
| content on _their_ packages.
| mwilliamson wrote:
| I don't know about other distros, but Debian makes it
| extremely easy to download both the binary package and
| the source package. For instance, on the page for the jq
| package [1], you can download the source using the links
| down the right-hand side, which includes the full test
| suite. The key, in my view, is that Debian has a nice way
| to associate both the final output artefact and the
| source (both the original source and their patches) with
| a specific version.
|
| [1] https://packages.debian.org/bookworm/jq
| hinkley wrote:
| My first thought was, "include a dev and prod version of
| the package" but that creates a ton of regression surface
| area for a feature that most people can't be bothered with
| anyway.
|
| It's easy enough to have things work in pre-prod and fail
| in prod without running slightly different code between
| them.
|
| I think there is a solution to this, but it's going to
| require that we change to something a lot more beefy than
| semver to define interface compatibility. Semver is a
| gentlemen's agreement to adhere to the Liskov Substitution
| Principle. We are none of us gentlemen, least of all when
| considered together.
| yellow_lead wrote:
| As tech improves and new libraries are created, limitations of
| the past are sometimes removed. From this post, I'm curious about
| other 'low hanging fruit' (if I may call it that) in open source.
|
| *I don't mean it's a simple endeavor but at least it's simple to
| describe.
| bhouston wrote:
| What to cut total NPM traffic by 50% or more overall? Easy:
|
| Create a shared Brotli dictionary (or zstandard or whatever)
| based on the top NPM packages by download bandwidth and then have
| all npm packages compressed using it.
|
| I think this can be done server side by npmjs.org, where NPM
| packages are recompressed in this fashion after upload using the
| shared dictionary, thus it is an optional feature and fully
| backwards compatible.
|
| Riffing on this idea because of this new chrome feature, which
| does this in a flexible fashion:
| https://chromestatus.com/feature/5124977788977152
|
| EDIT: This recompression of packages may be insecure as the
| digital signature of the package no longer aligns, but then the
| trick is to sign the contents of the package rather than package
| itself.
| Cthulhu_ wrote:
| I don't know how compression works; would a common library for
| Javascript save bandwidth? I'm just thinking that JS is a lot
| of repeated characters and keywords, and any browser or npm or
| node installation could be shipped with a (partial) dictionary.
|
| Although the bandwidth savings would only be the size of that
| dictionary.
|
| Still, I can't help but think there's ways to improve size &
| bandwidth usage by a lot, even besides using a different
| compression algorithm; a non-HTTP transfer method, for example.
| infogulch wrote:
| The bandwidth savings would be the size of the dictionary
| _per downloaded package_.
| shanemhansen wrote:
| It would be a fun test to run. But I'm not encouraged by the
| fact that the existing brotli dictionary already contains a
| bunch of javascript specific stuff:
|
| https://gist.github.com/klauspost/2900d5ba6f9b65d69c8e
|
| brotli literally already has a tokens for
| function/return/throw/indexOf(/.match/.length/etc.
|
| Also verify after decompress is not without tradeoffs. On one
| hand we have folks like github who can't change the version of
| zlib because people rely on identical .tar.gz.
| https://news.ycombinator.com/item?id=34586917
|
| On the other hand we have a whole lot of iffy stuff you can do
| to make programs decompressing content use large amounts of
| resources https://en.wikipedia.org/wiki/Zip_bomb which makes
| "decompress this potentially untrusted file so that I can
| validate it's safe to use" hard.
| hoten wrote:
| Signing the contents means you must unzip it to confirm its
| validity, which exposes users to the latent security bugs of
| their unarchival program (or a zip bomb).
| bhouston wrote:
| If recompressing it on the server breaks the chain of trust
| of the client provided zip, then one would need to upgrade
| npm on the creator's side of things so that they create both
| uploads themselves (legacy zip, and the shared-dictionary
| brotli package) and sign both themselves.
|
| Wait, does npm have digital signatures at all? I sort of
| assumed it did, but does it really?
| IshKebab wrote:
| 50%? I doubt that, unless your dictionary is _literally_ the
| top NPM packages, in which case you 're not saving much
| bandwidth by sending them all in advance...
|
| Maybe a small dictionary would save you 20% though.
| bhouston wrote:
| It is hard to judge ahead of time and yeah it depends on the
| dictionary size. It is hard to argue concretely about
| hypotheticals.
|
| It would be a fun experiment to figure out the compression
| ratio of say the top 1000 packages given a shared brotli
| dictionary of X size. Just keep increases the dictionary size
| until you see diminishing returns.
|
| My estimate is based on NPM packages contain a bunch of super
| stereotypical files that if they are used to create the
| dictionary likely result in amazing compression ratios:
| package.json, package-lock.json, CSS, Tailwind, Bootstrap
| .gitignore, LICENSE, .eslint, README, React/Vue/Angular
| code...
| kimi wrote:
| Who cares about how large a NPM library is? the issue is what you
| send to your clients, not how monstrous is what you download once
| on the integration server.
| bdcravens wrote:
| This assumes third-party code is never served directly from the
| node_modules folder (or via proxies like unpkg)
| Karellen wrote:
| > In real terms that means a saving of around 1MB. That doesn't
| sound like much, but at 4 million weekly downloads, that would
| save 4TB of bandwidth per week.
|
| Yeah, who cares about 4TB/week? What is this, the '90s?!
| kimi wrote:
| That's 6 Mbytes per second.
| taway1237 wrote:
| For a single package. And npm hosts a few more than that.
| dpcx wrote:
| That's a lot of data that you're not paying for...
| bhouston wrote:
| > Who cares about how large a NPM library is?
|
| NPMJS.org probably cares. And having smaller downloads for
| everyone else would speed things up a bit.
|
| BTW I saw this recently about shared brotli dictionaries for
| delivering JS, which is nice:
| https://chromestatus.com/feature/5124977788977152
| fabiospampinato wrote:
| I mean they care to some degree, if they _really_ cared
| presumably everything would be compressed with zstd and
| served to the more modern npm-cli installations, and npm-cli
| would refuse to upload binaries that are not explicitly
| allowlisted.
| justinsaccount wrote:
| If anyone really cared, they would fix the broken ecosystem
| that encourages people to redownload the same packages millions
| of times.
|
| Picked a popular package at random, webpack. npm says version
| 5.88.2 released 3 months ago has 5,992,398 downloads in the
| last 7 days.
|
| I don't know how anyone can look at that see it as anything
| other than a massive failure.
|
| Fast connections and free bandwidth have caused people to
| completely ignore the fact that every time some CI pipeline
| runs, npm goes off and downloads 100MB of dependencies.
| Dependencies that haven't changed since the pipeline last ran
| 30 seconds ago.
|
| npm could fix this by aggressively rate limiting clients that
| have already downloaded the same package multiple times, but I
| guess as long as the vc funding is paying the bandwidth bill
| it's not a problem, and those "millions of downloads" make you
| look good.
| paulddraper wrote:
| Elaborate, please.
|
| The vast majority of those are from CI on ephemeral cloud
| instances.
|
| Do you think CI should not be run?
|
| Or CI should be run, but not on ephemeral cloud instances?
|
| Or CI should be run on ephemeral cloud instances, but the
| packages should be cached using a separate service from
| npmjs.com (e.g. S3)? If so, what makes this other service
| preferable?
| justinsaccount wrote:
| > Or CI should be run on ephemeral cloud instances, but the
| packages should be cached using a separate service from
| npmjs.com (e.g. S3)? If so, what makes this other service
| preferable?
|
| Yes, you should vendor external dependencies.
|
| A build should ideally not require internet access to
| complete.
|
| People learned nothing from leftpad.
| paulddraper wrote:
| > A build should ideally not require internet access to
| complete.
|
| You've got a non-internet CI with non-internet source
| code repository with non-internet vendored dependencies??
|
| Technically possible, but I call BS.
| justinsaccount wrote:
| > You've got a non-internet CI with non-internet source
| code repository with non-internet vendored dependencies??
|
| Vendored dependencies are pulled down from an internal s3
| bucket (and cached locally) before the build starts, the
| rest of the build runs with no internet access.
|
| Look how nix does this, it's basically the same.
| CharlieDigital wrote:
| > Fast connections and free bandwidth have caused people to
| completely ignore the fact that every time some CI pipeline
| runs, npm goes off and downloads 100MB of dependencies.
| Dependencies that haven't changed since the pipeline last ran
| 30 seconds ago.
|
| Maybe it's just me, but I've always thought it was well known
| best practice to cache your deps[0].
|
| I'm pretty certain that this can be achieved with most CI/CD
| tools.
|
| https://docs.github.com/en/actions/using-
| workflows/caching-d...
| justinsaccount wrote:
| I've seen a lot of pipelines that simply don't bother. Or
| maybe they tried, but the caching isn't working and since
| the build works in the end no one notices an extra 30
| seconds.
| hoten wrote:
| Lots. See the "npm node_modules blackhole" meme. For one
| practical reason, confirming the quality of the code in
| node_modules is so impossible a task it just isn't even
| attempted. So people are shipping code they have 0 knowledge
| of. For a paranoia-fueled reason, even devDep packages run the
| risk of being harmful (malware in the build process).
| kimi wrote:
| Of all the reasons why the NPM ecosystem sucks, the
| compression of archives is but a side note.
| simonw wrote:
| The most interesting thing about zstandard is how easy it makes
| it to train custom compression dictionaries, which can provide
| massive improvements against certain types of data.
|
| I'd like to see how well a custom dictionary trained against a
| few hundred npm packages could work against arbitrary extra npm
| packages. My hunch is that there are a lot of patterns - both in
| JavaScript code and in the JSON and README conventions used in
| those packages - that could help achieve much better compression.
| mbb70 wrote:
| We had billions of Protobufs to store in Cassandra as byte
| blobs, using a zstd dictionary dramatically reduced storage
| size and improved latency over the built in compression. The
| complexity overhead of managing these dictionaries and making
| sure the client always has access to the right dictionary to
| decompress was non-trivial but well worth it.
|
| We looked at Brotli as well but decompression speed at
| acceptable ratio was the most important factor for us, that
| plus the far superior docs and evangelism sealed the deal for
| zstd.
| metadat wrote:
| What kind of additional gains (% wise) did you see with
| custom dictionaries compared to vanilla zstd?
| bhouston wrote:
| Depends on how much shared entropy your data has. Could you
| test this by trying to compress all your content into one
| stream (shared dictionary) compared to compressing it into
| separate streams (no shared dictionary)?
| bhouston wrote:
| We think similarly. :) I posted the same idea while you were
| typing yours: https://news.ycombinator.com/item?id=37755005
| onedognight wrote:
| Yann Collet went from project management, to programming his
| calculator, to creating zstd. This podcast detailing his journey
| in quite inspiring.
|
| From Project Management to Data Compression innovator:
| https://podcasts.apple.com/us/podcast/corecursive-coding-sto...
| [deleted]
| sfink wrote:
| > The first [caveat] is that lzbench isn't able to compress an
| entire directory like tar , so I opted to use lib/npm.js for this
| test.
|
| As opposed to... just using the npm-9.7.1.tar file that the other
| tests were just using (sorta barely internally to tar, but I
| don't think tar does any fancy streaming or anything if you're
| passing something via --use-compress-program, certainly nothing
| that would skew the results more than replacing all of npm.tar
| with just npm.js.)
|
| In my local install, npm.tar is 25MB. npm.js is 16KB. It may not
| change the final outcome, but the data in the article do not
| support the conclusion. I would strongly suggest tarring up the
| npm directory and rerunning lzbench.
| terrelln wrote:
| In parts (1) and (2) comparing the default setting of Zstd (level
| 3) against the default setting of Brotli (level 11) is a bit
| misleading. It shows Brotli compressing ~30% better than Zstd,
| but Brotli's default level is >100x slower than Zstd's default
| level. Zstd level 3 is expected to run at hundreds of MB/s, and
| Brotli level 11 is expected to run at ~2 MB/s. The compression
| speed is only 30% slower because that benchmark includes the time
| to tar the directory, which is likely more expensive than the
| compression itself. As @sfink already suggested, just running
| lzbench on the npm-9.7.1.tar would be a better benchmark.
|
| In part (3), because its running only on lib/npm.js which is
| 13KB, you are getting skewed results which aren't directly
| applicable to the compression of npm-9.7.1.tar. Brotli excels at
| compressing small Javascript files, as this is where its
| dictionary provides the most benefit. The benefits of the
| dictionary for a large tar file will be negligible.
|
| However, in the npm-9.7.1.tar scenario we still expect Brotli
| level 11 to produce slightly smaller files than Zstd level 19.
| Likely ~5% smaller. But we do expect Zstd to provide
| significantly faster decompression speed.
| terrelln wrote:
| That said, this is an interesting article, and I love to see
| people experimenting with modern compression algorithms for
| package management! There are a lot of easy wins in this space.
| tuatoru wrote:
| > The compression speed is only 30% slower because that
| benchmark includes the time to tar the directory, which is
| likely more expensive than the compression itself.
|
| Amdahl's Law in action.
| prollings wrote:
| Amdahl's law states that "the overall performance improvement
| gained by optimizing a single part of a system is limited by
| the fraction of time that the improved part is actually
| used".
|
| For anyone else who didn't know.
___________________________________________________________________
(page generated 2023-10-03 23:00 UTC)