[HN Gopher] We should all be using dependency cooldowns
___________________________________________________________________
We should all be using dependency cooldowns
Author : todsacerdoti
Score : 453 points
Date : 2025-11-21 14:50 UTC (1 days ago)
(HTM) web link (blog.yossarian.net)
(TXT) w3m dump (blog.yossarian.net)
| elevation wrote:
| You could do a lot of this with CI if you scheduled a job to
| fetch the most recent packages once a month and record a manifest
| with the current versions, then, if no security issues are
| reported before the end of the cooldown period, run integration
| tests against the new manifest. If no tests fail, automatically
| merge this update into the project.
|
| For projects with hundreds or thousands of active dependencies,
| the feed of security issues would be a real fire hose. You'd want
| to use an LLM to filter the security lists for relevance before
| bringing them to the attention of a developer.
|
| It would be more efficient to centralize this capability as a
| service so that 5000 companies aren't all paying for an LLM to
| analyze the same security reports. Perhaps it would be enough for
| someone to run a service like cooldown.pypi.org that served only
| the most vetted packages to everyone.
| AndrewDucker wrote:
| Why would you use an LLM rather than just sort by CVE score?
| ok123456 wrote:
| CVE scores are junk. Most CVEs are noise.
| jayd16 wrote:
| Doesn't this mean you're leaving yourself open to known
| vulnerabilities during that "cool down" time?
| jcalvinowens wrote:
| Yep. Not only vulnerabilities, but just bugs in general, which
| usually matter more than than vulnerabilities IMHO.
| bityard wrote:
| Do you believe new releases don't introduce new bugs?
| jcalvinowens wrote:
| Obviously. Every release introduces bugs. There's an
| inevitable positive correlation between the amount of code
| we write and the number of bugs we introduce, we just try
| to minimize it.
|
| The probability of introducing bugs is a function of the
| amount of development being done. Releasing less often
| doesn't change that. In fact, under that assumption,
| delaying releases _strictly increases_ the amount of time
| users are affected by the average bug.
|
| People who do this tell themselves the extra time allows
| them to catch more bugs. But in my experience that's a
| bedtime story, most bugs aren't noticed until after
| deployment anyway.
|
| That's completely orthogonal to slowly rolling out changes,
| btw.
| woodruffw wrote:
| To be clear, there's no reason why you can't update
| dependencies in advance of a cooldown period. The cooldown is
| an enforced policy that you can choose to override as needed.
|
| (This also doesn't apply to vulnerabilities per se, since
| known vulnerabilities typically aren't evaluated against
| cooldowns by tools like Dependabot.)
| jcalvinowens wrote:
| No you can't, the cooldown period is _started_ by the new
| upstream release. So if you follow this "rule" you're
| _guaranteed to be behind the latest upstream release._
| woodruffw wrote:
| I don't understand what you mean. The cooldown period is
| something _you_ decide to enforce; you can _always_
| override it. It 's your prerogative as a responsible
| engineer to decide the boundaries of policy enforcement.
| jcalvinowens wrote:
| I mean, you can do anything you want. But you're
| inventing a new definition of "cooldown" different than
| TFA...
| woodruffw wrote:
| I wrote TFA, so I can ensure you that this is what I
| meant :-)
|
| (Can you say more about what you found unclear in the
| post? The post definitely does not say "thou shall not
| update before the cooldown," the argument was that
| cooldowns are a great default. Engineers are
| fundamentally always expected to exercise discretion
| because, per the post, there's no single, sound, perfect
| solution to supply chain risks.)
| jcalvinowens wrote:
| > A "cooldown" is exactly what it sounds like: a window
| of time between when a dependency is published and when
| it's considered suitable for use.
|
| ^ This is what you wrote. I don't understand how that
| could possibly be interpreted any other way than I wrote
| above: an enforced delay on deploying the new code after
| upstream releases it.
|
| > The post definitely does not say "thou shall not update
| before the cooldown," the argument was that cooldowns are
| a great default
|
| Sorry, that is such a cop out. "I didn't actually mean
| you should do this, I mean you should consider if you
| should maybe do this and you are free to decide not to
| and don't argue with me if you disagree every case is
| different". Either take a stand or don't.
| woodruffw wrote:
| I think this is an overly tendentious reading. Nobody
| else seems to have gotten hung up on this, because they
| understand that it's a policy, not an immutable law of
| nature.
|
| The argument advanced in the post is IMO clear: cooldowns
| are a sensible default to have, and empirically seem to
| be effective at mitigating the risk of compromised
| dependencies. I thought I took sufficient pains to be
| clear that they're not a panacea.
| jcalvinowens wrote:
| I'm simply saying I think the policy you're proposing is
| bad. It is completely bizarre to me you're trying to
| frame that as a semantic argument.
| woodruffw wrote:
| I'm not saying it's a semantic argument. I'm saying that
| the policy isn't universal, whereas your argument appears
| to hinge on me _thinking_ that it is. But this seems to
| have run its course.
| jcalvinowens wrote:
| That's a semantic argument.
|
| Me saying your proposed policy is bad is in no way
| predicated on any assumption you intended it to be
| "universal". Quite the opposite: the last thing anybody
| needs at work is yet another poorly justified bullshit
| policy they have to constantly request an "exception" to
| to do their job...
| swatcoder wrote:
| No.
|
| A sane "cooldown" is just for _automated_ version updates
| relying on semantic versioning rules, which is a pretty
| questionable practice in the first place, but is indeed made a
| lot more safe this way.
|
| You can still _manually_ update your dependency versions when
| you learn that your code is exposed to some vulnerability that
| 's purportedly been fixed. It's no different than manually
| updating your dependency version when you learn that there's
| some implementation bug or performance cliff that was fixed.
|
| You might even still use an automated system to identify these
| kinds of "critical" updates and bring them to your attention,
| so that you can review them and can appropriately assume
| accountability for the choice to incorporate them early,
| bypassing the cooldown, if you believe that's the right thing
| to do.
|
| Putting in that effort, having the expertise to do so, and
| assuming that accountability is kind of your "job" as a
| developer or maintainer. You can't just automate and delegate
| everything if you want people to be able to trust what you
| share with them.
| jayd16 wrote:
| If you could understand the quality of updates you're pulling
| in, that solves the issue entirely. The point is that you
| can't.
|
| There's no reason to pretend we live in a world where
| everyone is manually combing through the source of every
| dependency update.
| zahlman wrote:
| > A sane "cooldown" is just for automated version updates
| relying on semantic versioning rules, which is a pretty
| questionable practice in the first place, but is indeed made
| a lot more safe this way.
|
| I'm reminded of how new Setuptools versions are able to cause
| problems for so many people, basically because install tools
| default to setting up isolated build environments using the
| latest version of whatever compatible build backend is
| specified (which in turn defaults to "any version of
| Setuptools"). qv. my LWN article
| https://lwn.net/Articles/1020576/ .
| astrobe_ wrote:
| TFA shows that most vulnerabilities have a "window of
| opportunity" smaller than one day. Are you anxious going on
| week-end because Friday evening a zero-day or a major bug could
| be made public?
| jayd16 wrote:
| Well then you agree that the answer is yes. At the end of the
| article a 14 day window is mentioned but not dismissed and
| does not mention the downsides.
| Havoc wrote:
| > we should all
|
| Except if everyone does it chance of malicious things being
| spotted in source also drops by virtue of less eyeballs
|
| Still helps though in cases where maintainer spot it etc
| smaudet wrote:
| > also drops by virtue of less eyeballs
|
| I don't think the people automatically updating and getting hit
| with the supply chain attack are also scanning the code, I
| don't think this will impact them much.
|
| If instead, updates are explicitly put on cooldowns, with the
| option of manually updating sooner, then there would be more
| eyeballs, not fewer, as people are more likely to investigate
| patch notes, etc., possibly even test in isolation...
| woodruffw wrote:
| (Author of the post.)
|
| The underlying premise here is that supply chain security
| vendors are _honest_ in their claims about proactively scanning
| (and effectively detecting + reporting) malicious and
| compromised packages. In other words, it 's not about eyeballs
| (I don't think people who automatically apply Dependabot bumps
| are categorically reading the code anyways), but about rigorous
| scanning and reporting.
| mkoubaa wrote:
| Maybe it's more about mileage (attacks that actually happen)
| than eyeballs.
| tjpnz wrote:
| You might read the source if something breaks but in a
| successful supply chain attack that's unlikely to happen. You
| push to production, go home for the evening and maybe get
| pinged about it by some automation in a few weeks.
| pico303 wrote:
| I agree. This "cooldown" approach seems antithetical to some
| basic tenants of security in the open source world, namely that
| more eyeballs makes for better code. If we all stop looking at
| or using the thing, are these security professionals really
| going to find the supply-chain problems for us in the thing,
| for free?
|
| Instead of a period where you don't use the new version,
| shouldn't we instead be promoting a best practice of not just
| blindly using a package or library in production? This
| "cooldown" should be a period of use in dev or QA environments
| while we take the time to investigate the libraries we use and
| their dependencies. I know this can be difficult in many
| languages and package managers, given the plethora of libraries
| and dependencies (I'm looking at you in particular JavaScript).
| But "it's hard" shouldn't really be a good excuse for our best
| efforts to maintain secure and stable applications.
| jcalvinowens wrote:
| I hate this. Delaying real bugfixes to achieve some nebulous
| poorly defined security benefit is just bad engineering.
| wrs wrote:
| Do you upgrade all your dependencies _every day_? If not, then
| there's no real difference in upgrading as if it were 7 days
| ago.
| jerlam wrote:
| Your CI/CD might be setup to upgrade all your dependencies on
| every build.
| wrs wrote:
| I've seen a lot of CI/CD setups and I've never seen that.
| If that were common practice, it would certainly simplify
| the package manager, since there would be no need for
| lockfiles!
| jerlam wrote:
| I didn't necessarily say they were good CI/CD practices.
| chuckadams wrote:
| I do see some CI running without lockfiles, and there's
| still a contingent that believes that libraries should
| never commit their lockfiles. It's a reasonably good idea
| to _test_ a configuration without the lockfile, since any
| user of your dependency is using _their_ lockfile that
| their local solver came up with, not yours, but this
| ought to be something you'd do alongside the tests using
| the lockfile. So locking down the CI environment is a
| good idea for that and many other reasons.
|
| Realistically, no one does full side-by-side tests with
| and without lockfiles, but it's a good idea to at least
| do a smoke test or two that way.
| ktpsns wrote:
| Unattended upgrades for server installations are very common.
| For instance, for Ubuntu/Debian this updates by default daily
| (source: https://documentation.ubuntu.com/server/how-
| to/software/auto...). No cooldown implemented, AFAIK.
|
| Of course we talk about OS security upgrades here, not
| library dependencies. But the attack vector is similar.
| jcalvinowens wrote:
| I upgrade all dependencies every time I deploy anything. If
| you don't, a zero day is going to bite you in the ass: that's
| the world we now live in.
|
| If upgrading like that scares you, your automated testing
| isn't good enough.
|
| On average, the most bug free Linux experience is to run the
| latest version of everything. I wasted much more time
| backporting bugfixes before I started doing that, than I have
| spent on new bugs since.
| smaudet wrote:
| > zero day is going to bite you in the ass
|
| Maybe your codebase is truly filled with code that is that
| riddled with flaws, but:
|
| 1) If so, updating will not save you from zero days, only
| from whatever bugs the developers have found.
|
| 2) Most updates are not zero day patches. They are as
| likely to (unintentionally) introduce zero days as they are
| to patch them.
|
| 3) In the case where a real issue is found, I can't imagine
| it isn't hard to use the aforementioned security vendors,
| and use their recommendations to force updates outside of a
| cooldown period.
| jcalvinowens wrote:
| My codebase runs on top of the same millions of lines of
| decades old system code that yours does. You don't seem
| to appreciate that :)
| smaudet wrote:
| If you mean operating system code, that is generally
| opaque, and not quite what the article is talking about
| (you don't use a dependency manager to install code that
| you have reviewed to perform operating system updates -
| you can, and that is fantastic for you, but not I imagine
| what you mean).
|
| Although, even for Operating Systems, cooldown periods on
| patches are not only a good thing, but something that
| e.g. a large org that can't afford downtime will employ
| (managing windows or linux software patches, e.g.). The
| reasoning is the same - updates have just as much chance
| to introduce bugs as fix them, and although you hope your
| OS vendor does adequate testing, _especially in the case
| where you cannot audit their code_ , you have to wait so
| that either some 3rd party security vendor can assess
| system safety, or you are able to perform adequate
| testing yourself.
| starburst wrote:
| Upgrading to new version can also introduce new exploits,
| no amount of tests can find those.
|
| Some of these can be short-lived, existing only on a minor
| patch and fixed on the next one promptly but you'll get it
| if you upgrade constantly on the latest blindly.
|
| There is always risks either way but latest version doesn't
| mean the "best" version, mistakes, errors happens,
| performance degradation, etc.
| jcalvinowens wrote:
| Personally, I choose to aggressively upgrade and engage
| with upstreams when I find problems, not to sit around
| waiting and hoping somebody will notice the bugs and fix
| them before they affect me :)
| jhatemyjob wrote:
| That sounds incredibly stressful.
| icehawk wrote:
| > I upgrade all dependencies every time I deploy anything.
| If you don't, a zero day is going to bite you in the ass:
| that's the world we now live in.
|
| I think you're using a different definition of zero day
| than what is standard. Any zero day vulnerability is not
| going to have a patch you can get with an update.
| jcalvinowens wrote:
| Zero days often get fixed sooner than seven days. If you
| wait seven days, you're pointlessly vulnerable.
| saurik wrote:
| Only if you already upgraded to the one with the bug in
| it, and then only if you ignore "this patch is actually
| different: read this notice and deploy it immediately".
| The argument is not "never update quickly": it is don't
| routinely deploy updates constantly that are not known to
| be high priority fixes.
| jcalvinowens wrote:
| > The argument is not "never update quickly": it is don't
| routinely deploy updates constantly that are not known to
| be high priority fixes.
|
| Yes. I'm saying that's wrong.
|
| The default should always be to upgrade to new upstream
| releases immediately. Only in exceptional cases should
| things be held back.
| saurik wrote:
| But that isn't what you said? ;P "f you wait seven days,
| you're pointlessly vulnerable." <- this is clearly a
| straw man, as no one is saying you'd wait seven days to
| deploy THAT patch... but, if some new configuration file
| feature is added, or it is ported to a new architecture
| you aren't using--aka, the 99.99% of patches--you don't
| deploy THOSE patches for a while (and I'd argue seven
| days is way way too small) until you get a feel that it
| isn't a supply chain attack (or what will become a zero
| day). Every now and then, someone tries to fix a serious
| bug... most of the time, you are just rolling the die on
| adding a new bug that someone can quickly find and
| exploit you using.
| jcalvinowens wrote:
| You're completely missing the point.
|
| > this is clearly a straw man, as no one is saying you'd
| wait seven days to deploy THAT patch...
|
| The policy being proposed is that upgrades are delayed.
| So in a company where that policy was enforced, I would
| be required to request an exception to the policy for
| your hypothetical patch.
|
| That's unacceptable for me. That's requiring me to do
| extra work for a nebulous poorly quantified security
| "benefit". It's a waste of my time and energy.
|
| I'm saying the whole policy is unjustified and should
| never be applied by default. At all. It's stupid. Its
| harmful for zero demonstrable benefit.
|
| I'm being blunt because you seem determined to somehow
| misconstrue what I'm saying as a nitpicky argument. I'm
| saying the whole policy is terrible and stupid. If it
| were forced on me by an employer, I would quit.
| Seriously.
| icehawk wrote:
| _Known vulnerabilities_ often get fixed sooner than seven
| days.
|
| You will _not_ know how long it takes to get a zero day
| fixed, because zero in "zero day" ends when the vendor
| is informed:
|
| > "A zero day vulnerability refers to an exploitable bug
| in software that is unknown to the vendor."
| midasz wrote:
| Renovate (dependabot equiv I think) creates PRs, I usually
| walk through them every morning or when there's a bit of
| downtime. Playing with the idea to automerge patches and
| maybe even minor updates but up until now it's not that hard
| to keep up.
| swatcoder wrote:
| The point is to apply a cooldown to your "dumb" and
| unaccountable automation, not to your own professional judgment
| as an engineer.
|
| If there's a bugfix or security patch that applies to how your
| application uses the dependency, then you review the changes,
| manually update your version if you feel comfortable with those
| changes, and accept responsibility for the intervention if it
| turns out you made a mistake and rushed in some malicious code.
|
| Meanwhile, most of the time, most changes pushed to
| dependencies are not even in the execution path of any given
| application that integration with them, and so don't need to be
| rushed in. And most others are "fixes" for issues that were
| apparently not presenting an eminent test failure or support
| crisis for your users and don't warrant being rushed in.
|
| There's not really a downside here, for any software that's
| actually being actively maintained by a responsible engineer.
| jcalvinowens wrote:
| You're not thinking about the system dependencies.
|
| > Meanwhile, most of the time, most changes pushed to
| dependencies are not even in the execution path of any given
| application that integration with them
|
| Sorry, this is really ignorant. You don't appreciate how much
| churn their is in things like the kernel and glibc, even in
| stable branches.
| swatcoder wrote:
| > You're not thinking about the system dependencies.
|
| You're correct, because it's completely neurotic to worry
| about phantom bugs that have no actual presence of mind but
| must absolutely positively be resolved as soon as a
| candidate fix has been pushed.
|
| If there's a zero day vulnerability that affects your
| system, which is a rare but real thing, you can be notified
| and bypass a cooldown system.
|
| Otherwise, you've presumably either adapted your workflow
| to work around a bug or you never even recognized one was
| there. Either way, waiting an extra <cooldown> before
| applying a fix isn't going to harm you, but it will dampen
| the much more dramatic risk of instability and supply chain
| vulnerabilities associated with being on the bleeding edge.
| jcalvinowens wrote:
| > You're correct, because it's completely neurotic to
| worry about phantom bugs that have no actual presence of
| mind but must absolutely positively be resolved as soon
| as a candidate fix has been pushed.
|
| Well, I've made a whole career out of fixing bugs like
| that. Just because you don't see them doesn't mean they
| don't exist.
|
| It is shockingly common to see systems bugs that don't
| trigger for a long time by luck, and then suddenly
| trigger out of the blue everywhere at once. Typically
| it's caused by innocuous changes in unrelated code, which
| is what makes it so nefarious.
|
| The most recent example I can think of was an
| uninitialized variable in some kernel code: hundreds of
| devices ran that code reliably for a year, but an
| innocuous change in the userland application made the
| device crash on startup almost 100% of the time.
|
| The fix had been in stable for _months_ , they just
| hadn't bothered to upgrade. If they had upgraded, they'd
| have never known the bug existed :)
|
| I can tell dozens of stories like that, which is why I
| feel so strongly about this.
| saurik wrote:
| If your internal process is willing to ship and deploy
| upgrades--whether to your code or that of a third party--
| without testing even so minimally to notice that they
| cause almost a 100% chance of crashing, you need the
| advice to slow down your updates more than anyone...
| jcalvinowens wrote:
| Obviously, they caught the bug and didn't deploy it.
|
| The point is that a change to a completely unrelated
| component caused a latent bug to make the device
| unusable, ended up delaying a release for weeks and
| causing them have to pay me a bunch of money to fix it
| for them.
|
| If they'd been applying upgrades, they would have never
| even known it existed, and all that trouble would have
| been avoided.
|
| I mean, I'm sort of working against my own interest here:
| arguably I should _want_ people to delay upgrades so I
| can get paid to backport things for them :)
| 33a wrote:
| A lot of security problems can be solved by moving slower.
| solarist wrote:
| All of security problems can be solved by not moving at all.
| kykat wrote:
| What about food security
| solarist wrote:
| If you don't eat food you won't get food poisoning.
| ksherlock wrote:
| The slow food movement encourages eating local foods and
| gardening, among other things, so it actually improves food
| security, for people who aren't food insecure.
| theoldgreybeard wrote:
| jokes on them I already have 10 year dependency cooldowns on the
| app I work on at work!
| cheschire wrote:
| Retirement Driven Development
| nitwit005 wrote:
| This did make me think of our app running on Java 8.
|
| Although, I suppose we've probably updated the patch version.
| marcosdumay wrote:
| There's always the one that requires Java 8, and the one that
| requires Java >= 11.
| layer8 wrote:
| People in this thread are worried that they are significantly
| vulnerable if they don't update right away. However, this is
| mostly not an issue in practice. A lot of software doesn't have
| continuous deployment, but instead has customer-side deployment
| of new releases, which follow a slower rhythm of several weeks or
| months, barring emergencies. They are fine. Most vulnerabilities
| that _aren 't_ supply-chain attacks are only exploitable under
| special circumstances anyway. The thing to do is to _monitor_
| your dependencies and their published vulnerabilities, and for
| critical vulnerabilities to assess whether your product is affect
| by it. Only then do you need to update that specific dependency
| right away.
| silvestrov wrote:
| I think the main question is: do your app get unknown input
| (i.e. controlled by other people).
|
| Browsers get a lot of unknown input, so they have to update
| often.
|
| A Weather app is likely to only get input from one specific
| site (controlled by the app developers), so it should be
| relatively safe.
| duped wrote:
| A million times this. You update a dependency when there are
| bug fixes or features that you need (and this includes patching
| vulnerabilities!). Those situations are rare. Otherwise you're
| just introducing risk into your system - and not that you're
| going to be caught in some dragnet supply chain attack, but
| that some dependency broke something you relied on by accident.
|
| Dependencies are good. Churn is bad.
| latentsea wrote:
| I'd rather have excellent quality test suites, and pipelines
| I can have a high degree of confidence in to catch these
| issues and mitigate the risk, than dependencies that
| basically never get updated and then become very difficult to
| do so, which just introduce different classes of risk.
| jbkgujklgui wrote:
| Vulnerabilities are not rare, the two most popular
| programming languages, javascript (node) and python, have
| 1000+ CVEs in their official docker images. I.e. in practice
| useless and shouldn't be used by anyone for anything.
| embedding-shape wrote:
| > for critical vulnerabilities to assess whether your product
| is affect by it. Only then do you need to update that specific
| dependency right away.
|
| This is indeed what's missing from the ecosystem at large.
| People seem to be under the impression that if a new release of
| software/library/OS/application is released, you need to move
| to it today. They don't seem to actually look through the
| changes, only doing that if anything breaks, and then proceed
| to upgrade because "why not" or "it'll only get harder in the
| future", neither which feel like solid choices considering the
| trade-offs.
|
| While we've seen to already have known that it introduces
| massive churn and unneeded work, it seems like we're waking up
| to the realization that it is a security tradeoff as well, to
| stay at the edge of version numbers. Sadly, not enough tooling
| seems to take this into account (yet?).
| hypeatei wrote:
| > Sadly, not enough tooling seems to take this into account
|
| Most tooling (e.g. Dependabot) allows you to set an interval
| between version checks. What more could be done on that front
| exactly? Devs can already choose to check less frequently.
| mirashii wrote:
| The check frequency isn't the problem, it's the latency
| between release and update. If a package was released 5
| minutes before dependabot runs and you still update to it,
| your lower frequency hasn't really done anything.
| hypeatei wrote:
| What are the chances of that, though? The same could
| happen if you wait X amount of days for the version to
| "mature" as well. A security issue could be found five
| minutes after you update.
|
| EDIT: Github supports this scenario too (as mentioned in
| the article):
|
| https://github.blog/changelog/2025-07-01-dependabot-
| supports...
|
| https://docs.github.com/en/code-
| security/dependabot/working-...
| mirashii wrote:
| > What are the chances of that, though?
|
| The whole premise of the article is that they're
| substantially lower, because some time for the ecosystem
| of dependency scanners and users to detect and report is
| better than none.
| tracnar wrote:
| You could use this funky tool from oss-rebuild which proxies
| registries so they return the state they were at a past date:
| https://github.com/google/oss-rebuild/tree/main/cmd/timewarp
| hoistbypetard wrote:
| That... looks like it would let me do this exactly the way
| I want to for npm and python. For my C++ stuff, it's all
| vendored and manual anyway.
|
| I had not seen that tool. Thanks for pointing it out.
| pas wrote:
| > "it'll only get harder in the future"
|
| that's generally true, no?
|
| of course waiting a few days/weeks should be the minimum
| unless there's a CVE (or equivalent) that's _applies_
| jerf wrote:
| I fought off the local imposition of Dependabot by executive
| fiat about a year ago by pointing out that it _maximizes_
| vulnerabilities to supply chain attacks if blindly followed
| or used as a metric excessively stupidly. Maximizing
| vulnerabilities was not the goal, after all. You do not want
| to harass teams with the fact that DeeplyNestedDepen just
| went from 1.1.54-rc2 to 1.1.54-rc3 because the worst case is
| that they upgrade just to shut the bot up.
|
| I think I wouldn't object to "Dependabot on a 2-week delay"
| as something that at least flags. However working in Go more
| than anything else it was often the case even so that
| dependency alerts were just an annoyance if they aren't tied
| to a security issue or something. Dynamic languages and
| static languages do not have the same risk profiles at all.
| The idea that some people have that all dependencies are
| super vital to update all the time and the casual expectation
| of a constant stream of vital security updates is not a
| general characteristic of programming, it is a _specific_
| characteristic not just of certain languages but arguably the
| community attached to those languages.
|
| (What we really need is capabilities, even at a very gross
| level, so we can all notice that the supposed vector math
| library suddenly at version 1.43.2 wants to add network
| access, disk reading, command execution, and cryptography to
| the set of things it wants to do, which would raise all sorts
| of eyebrows immediately, even perhaps in an automated
| fashion. But that's a separate discussion.)
| skybrian wrote:
| It seems like some of the arguments in favor of doing
| frequent releases apply at least a little bit for
| dependency updates?
|
| Doing updates on a regular basis (weekly to monthly) seems
| like a good idea so you don't forget how to do them and the
| work doesn't pile up. Also, it's easier to debug a problem
| when there are fewer changes at once.
|
| But they could be rescheduled depending on what else is
| going on.
| michaelcampbell wrote:
| > Doing updates on a regular basis (weekly to monthly)
|
| This lessens, but doesn't eliminate supply side vulns.
| You can still get a vulnerable new release if your
| schedule happens to land just after the vuln lands.
|
| TFA proposes a _delay_ in a particular dependency being
| pulled in. You can still update every
| day/hour/microsecond if you want, you just don't get the
| "new" thing until it's baked a bit.
| skybrian wrote:
| Yes, understood. Not arguing against cooldowns.
| catlifeonmars wrote:
| I use a dependabot config that buckets security updates
| into a separate pull than other updates. The non-security
| update PRs are just informational (can disable but I choose
| to leave them on), and you can actually spend the time to
| vet the security updates
| dudeinjapan wrote:
| Dependabot only suggest upgrades when there are CVEs, and
| even then it just alerts and raises PRs, it doesn't force
| it on you. Our team sees it as a convenience, not a
| draconian measure.
| dap wrote:
| At my last job, we only updated dependencies when there was a
| compelling reason. It was awful.
|
| What would happen from time to time was that an important
| reason _did_ come up, but the team was now many releases
| behind. Whoever was unlucky enough to sign up for the project
| that needed the updated dependency now had to do all those
| updates of the dependency, including figuring out how they
| affected a bunch of software that they weren 't otherwise
| going to work on. (e.g., for one code path, I need a bugfix
| that was shipped three years ago, but pulling that into my
| component affects many other code paths.) They now had to go
| figure out what would break, figure out how to test it, etc.
| Besides being awful for them, it creates bad incentives
| (don't sign up for those projects; put in hacks to avoid
| having to do the update), and it's also just plain bad for
| the business because it means almost any project, however
| simple it seems, might wind up running into this pit.
|
| I now think of it this way: either you're on the dependency's
| release train or you jump off. If you're on the train, you
| may as well stay pretty up to date. It doesn't need to be
| every release the minute it comes out, but nor should it be
| "I'll skip months of work and several major releases until
| something important comes out". So if you decline to update
| to a particular release, you've got to ask: am I jumping off
| forever, or am I just deferring work? If you think you're
| just deferring the _decision_ until you know if there 's a
| release worth updating to, you're really rolling the dice.
|
| (edit: The above experience was in Node.js. Every change in a
| dynamically typed language introduces a lot of risk. I'm now
| on a team that uses Rust, where knowing that the program
| compiles and passes all tests gives us a lot of confidence in
| the update. So although there's a lot of noise with regular
| dependency updates, it's not actually that much work.)
| JoshTriplett wrote:
| > I'm now on a team that uses Rust, where knowing that the
| program compiles and passes all tests gives us a lot of
| confidence in the update.
|
| That's been my experience as well. In addition, the
| ecosystem largely holds to semver, which means a non-major
| upgrade tends to be painless, and conversely, if there's a
| _major_ upgrade, you know not to put it off for too long
| because it 'll involve some degree of migration.
| darccio wrote:
| It's a similar experience in Go, specially because
| imports are done by URL and major versions higher than
| v1.x are forced to change it to add a suffix `/vN` at the
| end.
|
| Although this is true, any large ecosystem will have some
| popular packages not holding to semver properly. Also,
| the biggest downside is when your `>=v1` depends -
| indirectly usually - on a `v0` dependency which is
| allowed to do breaking changes.
| lock1 wrote:
| I think it also depends on the community as well. Last time
| I touched Node.js and Javascript-related things, every time
| I tried to update something, it practically guaranteed
| something would explode for no reason.
|
| While my recent legacy Java project migration from JDK 8 ->
| 21 & a ton of dependency upgrades has been a pretty smooth
| experience so far.
| Terr_ wrote:
| Yeah, along with any community's attitudes to risk and
| quality, there is also a varying, er, _chronological_
| component.
|
| I'd prefer to upgrade around the time most of the nasty
| surprises have already been discovered by somebody else,
| preferably with workarounds developed.
|
| At the same time, you don't want to be so far back that
| upgrading uncovers novel migration problems, or issues
| that nobody else cares about anymore.
| zdc1 wrote:
| Yeah, the JavaScript/Node.js ecosystem is pain. Lots of
| tooling (ORMs, queue/workflow frameworks, templating) is
| new-ish or quickly changing. I've also had minor updates
| cause breakages; semver is best-effort at best.
|
| I don't like Java but sometimes I envy their ecosystem.
| kakacik wrote:
| There is a reason most of stable companies use Java -
| _stability_. Outside of startups and SV, there are few
| reasons to avoid such a robust system.
|
| Plus you can find endless stream of experienced devs for
| it. Which are more stable job wise than those who come &
| go every 6-12 months. Stability. Top management barely
| cares for anything else from IT.
| coredog64 wrote:
| My current employer publishes "staleness" metrics at the
| project level. It's imperfect because it weights all the
| dependencies the same, but it's better than nothing.
| m000 wrote:
| I wonder, are there tools to help you automate this? I.e.
| to assign a value to the staleness of each package
| instead of simple "oudated" boolean, and also a weight to
| each package.
|
| E.g. something like: pkg_staleness = (1 *
| missed_patch + 5 * missed_minor + 10 * missed_major) *
| (month_diff(installed_release_date,
| latest_release_date)/2)^2 proj_staleness =
| sum(pkg.staleness * pkg.weight for pkg in all_packages)
| decasia wrote:
| My current org uses this:
|
| https://libyear.com/
| ozim wrote:
| Update at least quarterly so you don't have them stale an
| super hard to update
| teddyh wrote:
| You can choose to either live at the slightly-bleeding edge
| (as determined by "stable" releases, etc), or to live on
| the edge of end-of-life, as discussed here:
| <https://news.ycombinator.com/item?id=21785399>
| cncjchsue7 wrote:
| This is why I don't use dependencies that break backwards
| compatibility.
|
| If you break my code I'm not wasting time fixing what you
| broke, I'm fixing the root cause of the bug: finding your
| replacement.
| xp84 wrote:
| OP wisely said _for critical vulnerabilities_ is where the
| actual exposure needs to be assessed, _in order to make an
| exception_ from a rule like "install the latest release of
| things that's been published for X length of time."
|
| For instance if you use a package that provides a calendar
| widget and your app uses only the "western" calendar and
| there is a critical vulnerability that only manifests in
| the Islamic calendar, you have zero reason to worry about
| an exploit.
|
| I see this as a reasonable stance.
| samus wrote:
| That's indeed reasonable, but the opposite can happen
| just as well: there is a vulnerability in the western
| calendar, but I'm on an old major.minor version that
| receives no security patches anymore. So now I have to
| upgrade that dependency, potentially triggering an
| avalanche of incompatibilities with other packages,
| leading to further upgrades and associated breakages.
| Oopsie.
| iainmerrick wrote:
| Yes, I've had exactly the same experience. Once you get off
| the dependency train, it's almost impossible to get back
| on.
|
| I don't think this is specific to any one language or
| environment, it just gets more difficult the larger your
| project is and the longer you go without updating
| dependencies.
|
| I've experienced this with NPM projects, with Android
| projects, and with C++ (neglecting to merge upstream
| changes from a private fork).
|
| It does seem likely that dynamic languages make this
| problem worse, but I don't think very strict statically
| typed languages completely avoid it.
| stefan_ wrote:
| Thats because the security industry has been captured by
| useless middle manager types who can see that "one dependency
| has a critical vulnerability", but could never in their life
| scrounge together the clue to analyze the impact of that
| vulnerability correctly. All they know is the checklist
| fails, and the checklist can not fail.
|
| (Literally at one place we built a SPA frontend that was
| embedded in the device firmware as a static bundle, served to
| the client and would then talk to a small API server. And
| because these NodeJS types liked to have libraries reused for
| server and frontend, we would get endless "vulnerability
| reports" - but all of this stuff only ever ran in the clients
| browser!)
| patrick451 wrote:
| Jumping straight to the new release because it fixed one
| security bug has always struck me as a round about way of
| trying to achieve security through obscurity, especially when
| the releases include tons of other changes. Yes, this release
| fixed CVE-123, but how many new ones were added?
| jazzyjackson wrote:
| This is a valid security strategy tho, always shifting the
| ground beneath the attackers feet. As the code author, you
| might not know where there are vulnerabilities in your
| code, but someone targeting you does. You will never have
| bug free code, so better to just keep it in constant flux
| than allow an attacker to analyze an unchanging application
| over months and years.
| jerf wrote:
| Also, if you are updating "right away" it is presumably because
| of some specific vulnerability (or set of them). But if you're
| in an "update right now" mode you have the most eyes on the
| source code in question at that point in time, and it's
| probably a relatively small patch for the targeted problem.
| Such a patch is the absolute worst time for an attacker to try
| to sneak anything in to a release, the exact and complete
| opposite of the conditions they are looking for.
|
| Nobody is proposing a system that utterly and completely locks
| you out of all updates if they haven't aged enough. There is
| always going to be an override switch.
| justsomehnguy wrote:
| > People in this thread are worried that they are significantly
| vulnerable if they don't update right away
|
| Most of them assume what if they are working on some public
| accessible website then 99% of the people and orgs in the world
| are running nothing but some public accessible website.
| bumblehean wrote:
| >The thing to do is to monitor your dependencies and their
| published vulnerabilities, and for critical vulnerabilities to
| assess whether your product is affect by it. Only then do you
| need to update that specific dependency right away.
|
| The practical problem with this is that many large
| organizations have a security/infosec team that mandates a
| "zero CVE" posture for all software.
|
| Where I work, if our infosec team's scanner detect a critical
| vulnerability in any software we use, we have 7 days to update
| it. If we miss that window we're "out of compliance" which
| triggers a whole process that no one wants to deal with.
|
| The path of least resistance is to update everything as soon as
| updates are available. Consequences be damned.
| BrenBarn wrote:
| > The practical problem with this is that many large
| organizations have a security/infosec team that mandates a
| "zero CVE" posture for all software.
|
| The solution is to fire those teams.
| bumblehean wrote:
| Sure I'll go suggest that to my C-suite lol
| paulddraper wrote:
| Someone has the authority to fix the problem. Maybe it's
| them.
| acdha wrote:
| This isn't a serious response. Even if you had the clout to
| do that, you'd then own having to deal with the underlying
| pressure which lead them to require that in the first
| place. It's rare that this is someone waking up in the
| morning and deciding to be insufferable, although you can't
| rule that out in infosec, but they're usually responding to
| requirements added by customers, auditors needed to get
| some kind of compliance status, etc.
|
| What you should do instead is talk with them about SLAs and
| validation. For example, commit to patching CRITICAL within
| x days, HIGH with y, etc. but also have a process where
| those can be cancelled if the bug can be shown not to be
| exploitable in your environment. Your CISO should be
| talking about the risk of supply chain attacks and outages
| caused by rushed updates, too, since the latter are pretty
| common.
| IcyWindows wrote:
| Aren't some of these government regulations for cloud,
| etc.?
| the8472 wrote:
| There are several layers of translation between public
| regulations, a company's internal security policy and the
| processes used to enforce those policies.
|
| Let's say the reg says you're liable for damages caused
| by software defects you ship due to negligence, giving
| you broad leeway how to mitigate risks. The corporate
| policy then says "CVEs with score X must be fixed in Y
| days; OWASP best practices; infrastructure audits; MFA;
| yadda yadda". Finally the enforcement is then done by
| automated tooling like sonarqube, prisma. dependabot,
| burpsuite, ... and any finding must be fixed with little
| nuance because the people doing the scans lack the time
| or expertise to assess whether any particular finding is
| actually security-relevant.
|
| On the ground the automated, inflexible enforcement and
| friction then leads to devs choosing approaches that
| won't show up in scans, not necessarily secure ones.
|
| As an example I witnessed recently: A cloud infra
| scanning tool highlighted that an AppGateway was used as
| TLS-terminating reverse proxy, meaning it used HTTP
| _internally_. The tool says "HTTP bad", even when it's
| on an isolated private subnet. But the tool didn't
| understand Kubernetes clusters, so a a public unencrypted
| ingress, i.e. _public_ HTTP didn 't show up. The former
| was treated as a critical issue that must be fixed asap
| or the issue will get escalated up the management chain.
| The latter? Nobody cares.
|
| Another time I got pressure to downgrade from Argon2 to
| SHA2 for password hashing because Argon2 wasn't on their
| whitelist. I resisted that change but it was a stressful
| bureaucratic process with some leadership being very
| unhelpful and suggesting "can't you just do the compliant
| thing and stop spending time on this?".
|
| So I agree with GP that some security teams barely
| correlate with security, sometimes going into the
| negative. A better approach would to integrate software
| security engineers into dev teams, but that'd be more
| expensive and less measurable than "tool says zero CVEs".
| swiftcoder wrote:
| SOC2/FIPS/HIPAA/etc don't mandate zero-CVE, but a zero-
| CVE posture is an easy way to dodge all the paperwork
| that would be involved in exhaustively documenting
| exactly why each flagged CVE doesn't actually apply to
| your specific scenario (and then potentially re-
| litigating it all again in your annual audit).
|
| So it's more of a cost-cutting/cover-your-ass measure
| than an actual requirement.
| tetha wrote:
| I really dislike that approach. We're by now evaluating high-
| severity CVEs ASAP in a group to figure out if we are
| affected, and if mitigations apply. Then there is the choice
| of crash-patching and/or mitigating in parallel, updating
| fast, or just prioritizing that update more.
|
| We had like 1 or 2 crash-patches in the past - Log4Shell was
| one of them, and blocking an API no matter what in a
| component was another one.
|
| In a lot of other cases, you could easily wait a week or two
| for directly customer facing things.
| hiAndrewQuinn wrote:
| You view this as a burden, but (at least if you operate in
| the EU) I'd argue you're actually looking at a competitive
| advantage that hasn't cashed out yet.
|
| Come 2027-12, the Cyber Resilience Act enters full
| enforcement. The CRA mandates a "duty of care" for the
| product's lifecycle, meaning if a company blindly updates a
| dependency to clear a dashboard and ships a supply-chain
| compromise, they are on the hook for fines up to EUR15M or
| 2.5% of global turnover.
|
| At that point, yes, there is a sense in which the blind
| update strategy you described becomes a legal liability. But
| don't miss the forest for the trees, here. Most software
| companies are doing zero vetting whatsoever. They're staring
| at the comet tail of an oncoming mass extinction event. The
| fact that you are already thinking in terms of "assess
| impact" vs. "blindly patch" already puts your workplace
| significantly ahead of the market.
| rcxdude wrote:
| The CRA, unfortunately, also has language along the lines
| of "don't ship with known vulnerabilities", without
| defining who determines what is a vulnerability and how, so
| I fully expect this no-thoughts-only-checkboxes approach to
| increase with it (there's already a bunch of other
| standards which can be imposed on organizations from
| various angles which essentially force updates without any
| consideration of the risk of introducing new
| vulnerabilities or supply-chain attacks).
| latentsea wrote:
| My previous job we did continuous deployment and had a
| weekly JIRA ticket where an engineer would merge dependabot
| PRs. We scanned everything in our stack with Trivy to be
| aware of security vulnerabilities and had processes to
| ensure they were always patched within 2 weeks.
| ozim wrote:
| Fun part is people are worried about 0 days but in reality most
| problems come from 300 or 600 days old not patched vulns.
| nrhrjrjrjtntbt wrote:
| I agree. Pick yout poison. my poison is waiting before
| upgrades, assess zero days case by case.
| weinzierl wrote:
| _" The thing to do is to monitor your dependencies and their
| published vulnerabilities, and for critical vulnerabilities to
| assess whether your product is affect by it."_
|
| Yes
|
| _" Only then do you need to update that specific dependency
| right away."_
|
| Big no. If you do that it is guaranteed one day you miss a
| vulnerability that hurts you.
|
| To frame it differently: What you propose sounds good in theory
| but in practice the effort to evaluate vulnerabilities against
| your product will be higher than the effort to update plus
| taking appropriate measures against supply chain attacks.
| cosmic_cheese wrote:
| I think there's a much stronger argument for policies that both
| limit the number and complexity of dependencies. Don't add it
| unless it's highly focused (no "everything libraries" that pull
| in entire universes of their own) and carries a high level of
| value. A project's entire dependency tree should be small and
| clean.
|
| Libraries themselves should perhaps also take a page from the
| book of Linux distributions and offer LTS (long term support)
| releases that are feature frozen and include only security
| patches, which are much easier to reason about and periodically
| audit.
| pengaru wrote:
| I'd be willing to pay $100 to upvote your comment 100x.
| Y_Y wrote:
| How do you think dang puts bread on the table?
| tcfhgj wrote:
| Won't using highly focused dependencies increase the amount of
| dependencies?
|
| Limiting the number of dependencies, but then rewriting them in
| your own code, will also increase the maintenance burden and
| compile times
| skydhash wrote:
| A lot of projects are using dependencies, but are only using
| a small part of. Or are using them in a single place for a
| single usecase. Like bringing in formik (npm), but you only
| have one single form. Or moment, because you want to format a
| single date.
| buu700 wrote:
| I think AI nudges the economics more in this direction as well.
| Adding a non-core dependency has historically bought short-term
| velocity in exchange for different long-term maintenance costs.
| With AI, there are now many more cases where a first-party
| implementation becomes cheaper/easier/faster in both the short
| term and the long term.
|
| Of course it's up to developers to weigh the tradeoffs and make
| reasonable choices, but now we have a lot more optionality.
| Reaching for a dependency no longer needs to be the default
| choice of a developer on a tight timeline/budget.
| xmodem wrote:
| Let's have AI generate the same vulnerable code across
| hundreds of projects, most of which will remain vulnerable
| forever, instead of having those projects all depend on a
| central copy of that code that can be fixed and distributed
| once the issue gets discovered. Great plan!
| buu700 wrote:
| You're attacking a straw man. No one said not to use
| dependencies.
| xmodem wrote:
| At one stage in my career the startup I was working at
| was being acquired, and I was conscripted into the due-
| diligence effort. An external auditor had run a scanning
| tool over all of our repos and the team I was on was
| tasked with going through thousands of snippets across
| ~100 services and doing _something_ about them.
|
| In many cases I was able to replace 10s of lines of code
| with a single function call to a dependency the project
| already had. In very few cases did I have to add a new
| dependency.
|
| But directly relevant to this discussion is the story of
| the most copied code snippet on stack overflow of all
| time [1]. Turns out, it was buggy. And we had more than
| once copy of it. If it hadn't been for the due diligence
| effort I'm 100% certain they would still be there.
|
| [1]: https://news.ycombinator.com/item?id=37674139
| buu700 wrote:
| Sure, but that doesn't contradict the case for
| conservatism in adding new dependencies. A maximally
| liberal approach is just as bad as the inverse. For
| example:
|
| * Introducing a library with two GitHub stars from an
| unknown developer
|
| * Introducing a library that was last updated a decade
| ago
|
| * Introducing a library with a list of aging unresolved
| CVEs
|
| * Pulling in a million lines of code that you're
| reasonably confident you'll never have a use for 99% of
|
| * Relying on an insufficiently stable API relative to the
| team's budget, which risks eventually becoming an
| obstacle to applying future security updates (if you're
| stuck on version 11.22.63 of a library with a current
| release of 20.2.5, you have a problem)
|
| Each line of code included is a liability, regardless of
| whether that code is first-party or third-party. Each
| dependency in and of itself is also a liability and
| ongoing cost center.
|
| Using AI doesn't magically make all first-party code
| insecure. Writing good code and following best practices
| around reviewing and testing is important regardless of
| whether you use AI. The point is that AI reduces the
| upfront cost of first-party code, thus diluting the
| incentive to make short-sighted dependency management
| choices.
| xmodem wrote:
| > Introducing a library with two GitHub stars from an
| unknown developer
|
| I'd still rather have the original than the AI's un-
| attributed regurgitation. Of course the fewer users
| something has, the more scrutiny it requires, and below a
| certain threshold I will be sure to specify an exact
| version and leave a comment for the person bumping deps
| in the future to take care with these.
|
| > Introducing a library that was last updated a decade
| ago
|
| Here I'm mostly with you, if only because I will likely
| want to apply whatever modernisations were not possible
| in the language a decade ago. On the other hand, if it
| has been working without updates in a decade, and people
| are STILL using it, that sounds pretty damn battle-
| hardened by this point.
|
| > Introducing a library with a list of aging unresolved
| CVEs
|
| How common is this in practice? I don't think I've ever
| gone library hunting and found myself with a choice
| between "use a thing with unsolved CVEs" and "rewrite it
| myself". Normally the way projects end up depending on
| libraries with lists of unresolved CVEs is by adopting a
| library that subsequently becomes unmaintained. Obviously
| this is a painful situation to be in, but I'm not sure
| its worse than if you had replicated the code instead.
|
| > Pulling in a million lines of code that you're
| reasonably confident you'll never have a use for 99% of
|
| It very much depends - not all imported-and-unused code
| is equal. Like yeah, if you have Flask for your web
| framework, SQLAlchemy for your ORM, Jinja for your
| templates, well you probably shouldn't pull in Django for
| your authentication system. On the other hand, I would be
| shocked if I had ever used more than 5% of the standard
| library in the languages I work with regularly. I am
| definitely NOT about to start writing my rust as no_std
| though.
|
| > Relying on an insufficiently stable API relative to the
| team's budget, which risks eventually becoming an
| obstacle to applying future security updates (if you're
| stuck on version 11.22.63 of a library with a current
| release of 20.2.5, you have a problem)
|
| If a team does not have the resources to keep up to date
| with their maintenance work, that's a problem. A problem
| that is far too common, and a situation that is unlikely
| to be improved by that team replicating the parts of the
| library they need into their own codebase. In my
| experience, "this dependency has a CVE and the security
| team is forcing us to update" can be one of the few ways
| to get leadership to care about maintenance work at all
| for teams in this situation.
|
| > Each line of code included is a liability, regardless
| of whether that code is first-party or third-party. Each
| dependency in and of itself is also a liability and
| ongoing cost center.
|
| First-party code is an individual liability. Third-party
| code can be a shared one.
| buu700 wrote:
| _> I 'd still rather have the original than the AI's un-
| attributed regurgitation._
|
| If what you need happens to be exactly what the library
| provides -- nothing more, less, or different -- then I
| see where you're coming from. The drawback is that the
| dependency itself remains a liability. With such an
| obscure library, you'll have fewer eyes watching for
| supply chain attacks.
|
| The other issues are that 1) an obscure library is more
| likely to suddenly become unmaintained; and 2) someone on
| the team has to remember to include it in scope of
| internal code audits, since it may be receiving little or
| no other such attention.
|
| _> On the other hand, I would be shocked if I had ever
| used more than 5% of the standard library in the
| languages I work with regularly._
|
| Hence "non-core". A robust stdlib or framework is in line
| with what I'm suggesting, not a counterexample. I'm not
| anti-dependency, just being practical.
|
| My point is that AI gives developers more freedom to
| implement more optimal dependency management strategies,
| and that's a good thing.
|
| _> unlikely to be improved by that team replicating the
| parts of the library they need into their own codebase_
|
| At no point have I advised copying code from libraries
| instead of importing them.
|
| If you can implement a UI component that does exactly
| what you want and looks exactly how you want it to look
| in 200 lines of JSX with no dependencies, and you can
| generate and review the code in less than five minutes,
| why would you prefer to install a sprawling UI framework
| with one component that does something kind of similar
| that you'll still need to heavily customize? The latter
| won't even save you upfront time anymore, and in exchange
| you're signing up for years of breaking changes and
| occasional regressions. That's the best case scenario;
| worst case scenario it's suddenly deprecated or abandoned
| and you're no longer getting security updates.
|
| It seems like you're taking a very black-and-white view
| in favor of outsourcing to dependencies. As with
| everything, there are tradeoffs that should be weighed on
| a case-by-case basis.
| xmodem wrote:
| I've seen this argument made frequently. It's clearly a popular
| sentiment, but I can't help feel that it's one of those things
| that sounds nice in theory if you don't think about it too
| hard. (Also, cards on the table, I personally really _like_
| being able to pull in a tried-and-tested implementation of code
| to solve a common problem that 's also used by in some cases
| literally millions of other projects. I dislike having to re-
| solve the same problem I have already solved elsewhere.)
|
| Can you cite an example of a moderately-widely-used open source
| project or library that is pulling in code as a dependency that
| you feel it should have replicated itself?
|
| What are some examples of "everything libraries" that you view
| as problematic?
| skydhash wrote:
| Anything that pulled in chalk. You need a very good reason to
| emit escape sequences. The whole npm (and rust, python,..)
| ecosystem assumes that if it's a tty, then it's a full blown
| xterm-256color terminal. And then you need to pipe to cat or
| less to have sensible output.
|
| So if you're adding chalk, that generally means you don't
| know jack about terminals.
| igregoryca wrote:
| Some people appreciate it when terminal output is easier to
| read.
|
| If chalk emits sequences that aren't supported by your
| terminal, then that's a deficiency in chalk, not the
| programs that wanted to produce colored output. It's easier
| to fix chalk than to fix 50,000 separate would-be
| dependents of chalk.
| Dylan16807 wrote:
| I appreciate your frustration but this isn't an answer to
| the question. The question is about implementing the same
| feature in two different ways, dependency or internal code.
| Whether a feature _should_ be added is a different
| question.
| zahlman wrote:
| In the Python world, people often enough use Rich so that
| they can put codes like [red] into a string that are
| translated into the corresponding ANSI. The end user pays
| several megabytes for this by default, as Rich will also
| pull in Pygments, which is basically a collection of lexers
| for various programming languages to enable syntax
| highlighting. They also pay for a rather large database of
| emoji names, a Markdown parser, logic for table generation
| and column formatting etc. all of which might go unused by
| someone who just doesn't want to remember \e[31m (or re-
| create the lookup table and substitution code).
| joquarky wrote:
| Exactly! ANSI escape codes are old and well defined for
| all the basic purposes.
|
| Pulling in a huge library just to set some colors is like
| hiring a team of electrical contractors to plug in a
| single toaster.
| joquarky wrote:
| Chalk appears to be a great example.
|
| I wonder how many devs are pulling in a whole library just
| to add colors. ANSI escape sequences are as old as dirt and
| very simple.
|
| Just make some consts for each sequence that you intend to
| use. That's what I do, and it typically only adds a dozen
| or so lines of code.
| ted_dunning wrote:
| The problem isn't the implementation of what I _want_ to do.
| It 's all of the implementations of things I never cared
| about doing. And the implementation of what I want to do that
| is soooo much more complex than it needs to be that I could
| easily have implemented it myself in less time.
|
| The problem is also less about the implementation I want,
| it's about the 10,000 dependencies of things I don't really
| want. All of those are attack surface much larger than some
| simple function.
| igregoryca wrote:
| Most of your supply chain attack surface is social engineering
| attack surface. Doesn't really matter if I use Lodash, or 20
| different single-function libraries, if I end up trusting the
| exact same people to not backdoor my server.
|
| Of course, small libraries get a bad rap because they're often
| maintained by tons of different people, especially in less
| centralized ecosystems like npm. That's usually a fair
| assessment. But a single author will sometimes maintain 5, 10,
| or 20 different popular libraries, and adding another library
| of theirs won't really increase your social attack surface.
|
| So you're right about "pull[ing] in universes [of package
| maintainers]". I just don't think complexity or number of
| packages are the metrics we should be optimizing. They are
| correlates, though.
|
| (And more complex code can certainly contain more
| vulnerabilities, but that can be dealt with in the traditional
| ways. Complexity begets simplicity, yadda yadda; complexity
| that only begets complexity should obviously be eliminated)
| mkoubaa wrote:
| The lower level the dependency is, the more unjustifiable it is
| for it to have its own dependencies. This ought to be a point
| of competition between libraries and often is, at least in the
| c++ world
| gr4vityWall wrote:
| The Debian stable model of having a distro handle common
| dependencies with a full system upgrade every few years looks
| more and more sane as years pass.
|
| It's a shame some ecosystems move waaay too fast, or don't have a
| good story for having distro-specific packages. For example, I
| don't think there are Node.js libraries packaged for Debian that
| allow you to install them from apt and use it in projects. I
| might be wrong.
| kykat wrote:
| It is possible to work with rust, using debian repositories as
| the only source.
| embedding-shape wrote:
| > For example, I don't think there are Node.js libraries
| packaged for Debian that allow you to install them from apt and
| use it in projects
|
| Web search shows some:
| https://packages.debian.org/search?keywords=node&searchon=na...
| (but also shows "for optimizing reasons some results might have
| been suppressed" so might not be all)
|
| Although probably different from other distros, Arch for
| example seems to have none.
| o11c wrote:
| Locally, you can do: apt-cache showpkg
| 'node-*' | grep ^Package:
|
| which returns 4155 results, though 727 of them are type
| packages.
|
| Using these in commonjs code is trivial; they are
| automatically found by `require`. Unfortunately, system-
| installed packages are yet another casualty of the ESM
| transition ... there are ways to make it work but it's not
| automatic like it used to be.
| 9dev wrote:
| > Unfortunately, system-installed packages are yet another
| casualty of the ESM transition ...
|
| A small price to pay for the abundant benefits ESM brings.
| o11c wrote:
| Honestly, I don't see the value gain, given how many
| other problems the JS tooling has (even ignoring the
| ecosystem).
|
| In particular, the fact that Typescript makes it _very_
| difficult to write a project that uses both browser-
| specific and node-specific functionality is particularly
| damning.
| noosphr wrote:
| Never mistake motion for action.
|
| An eco system moving too quickly, when it isn't being
| fundamentally changed, isn't a sign of a healthy ecosystem, but
| of a pathological one.
|
| No one can think that js has progressed substantially in the
| last three years, yet trying to build any project three years
| old without updates is so hard a rewrite is a reasonable
| solution.
| gr4vityWall wrote:
| > No one can think that js has progressed substantially in
| the last three years
|
| Are we talking about the language, or the wider ecosystem?
|
| If the latter, I think a lot of people would disagree. Bun is
| about three years old.
|
| Other significant changes are Node.js being able to run
| TypeScript files without any optional flags, or being able to
| use require on ES Modules. I see positive changes in the
| ecosystem in recent years.
| noosphr wrote:
| That is motion not action.
|
| The point of javascript is to display websites in the
| browser.
|
| Ask yourself, in the last three years has there been a
| substantial improvement in the way you access websites? Or
| have they gotten even slower, buggier and more annoying to
| deal with?
| FridgeSeal wrote:
| No but the devs can push slower, buggier, more annoying
| websites to prod, faster!
|
| And after all, isn't developer velocity (and investor
| benefits) really the only things that matter???
|
| /sssss
| throwawayqqq11 wrote:
| > modern websites
|
| Your are comparing the js ecosystem and bad project
| realizations/designs.
|
| > Action vs motion
|
| I think the main difference you mean is the motivation
| behind changes, is it a (re)action to achieve a
| meassurable goal, is this a fix for a critical CVE, or
| just some dev having fun and pumping up the numbers.
|
| GP mentioned the recent feature of executing ts, which is
| a reasonable goal imo, with alot of beneficial effects
| down the line but in the present just another hustle to
| take care about. So is this a pointless motion or a
| worthy action? Both statements can be correct, depending
| on your goals.
| gr4vityWall wrote:
| > The point of javascript is to display websites in the
| browser.
|
| I don't follow. JavaScript is a dynamic general purpose
| programming language. It is certainly not limited to
| displaying websites, nor it's a requirement for that. The
| improvements I mentioned in the previous post aren't
| things you'd get the benefit of inside a web browser.
| mkoubaa wrote:
| The stable model usually implies that your app has to target
| both the old and the new distro version for a while. That is a
| bit too much to ask for some, unfortunately
| l9o wrote:
| I wonder if LLMs can help here to some extent. I agree with
| others that cooldowns aren't helpful if everyone is doing it.
|
| I've been working on automatic updates for some of my [very
| overengineered] homelab infra and one thing that I've found
| particularly helpful is to generate PRs with reasonable summaries
| of the updates with an LLM. it basically works by having a script
| that spews out diffs of any locks that were updated in my
| repository, while also computing things like `nix store diff-
| closures` for the before/after derivations. once I have those
| diffs, I feed them into claude code in my CI job, which generates
| a pull request with a nicely formatted output.
|
| one thing I've been thinking is to lookup all of those
| dependencies that were upgraded and have the LLM review the
| commits. often claude already seems to lookup some of the commits
| itself and be able to give a high level summary of the changes,
| but only for small dependencies where the commit hash and
| repository were in the lock file.
|
| it would likely not help at all with the xz utils backdoor, as
| IIRC the backdoor wasn't even in the git repo, but on the release
| tarballs. but I wonder if anyone is exploring this yet?
| robszumski wrote:
| We built and launched this product about 2 months ago, HN
| thread here: https://news.ycombinator.com/item?id=45439721
|
| Totally agree that AI is great for this, it will work harder
| and go deeper and never gets tired of reading code or release
| notes or migration guides. What you want instead of summaries
| is to find the breaking changes, figure out if they matter,
| then comment on _that_.
| perlgeek wrote:
| Some scattered thoughts on that:
|
| * If everybody does it, it won't work so well
|
| * I've seen cases where folks pinned their dependencies, and then
| used "npm install" instead of "npm ci", so the pinning was
| worthless. Guess they are the accidental, free beta testers for
| the rest of us.
|
| * In some ecosystems, distributions (such as Debian) does both
| additional QA, and also apply a cooldown. Now we try to retrofit
| some of that into our package managers.
| exasperaited wrote:
| > * If everybody does it, it won't work so well
|
| Indeed, this is a complex problem to solve.
|
| And the "it won't work so well" of this is probably a general
| chilling effect on trying to fix things because people won't
| roll them out fast enough anyway.
|
| This may seem theoretical but for example in websites where
| there are suppliers and customers, there's quite a chilling
| effect on any mechanism that encourages people to wait until a
| supplier has positive feedback; there are fewer and fewer
| people with low enough stakes who are willing to be early
| adopters in that situation.
|
| What this means is that new suppliers often drop out too
| quickly, abandon platforms, work around those measures in a way
| that reduces the value of trust, and worse still there's a risk
| of bad reviews because of the reviewer's Dunning-Kruger etc.
|
| I think the mechanism is important for people who really _must_
| use it, but there will absolutely be side effects that are hard
| to qualify /correct.
| layer8 wrote:
| "If everybody does it, <some outcome>" is rarely a good
| argument, because the premise rarely becomes reality.
| dugidugout wrote:
| If you're prescribing a public practice you intend to be
| taken seriously, you should contend with what happens at
| scale, especially when evaluating competing alternatives,
| where comparative scale effects become decisive. Given the
| article's push for ecosystem-wide support, there's good
| reason to probe this hypothetical.
| rmunn wrote:
| So true.
|
| Things that everybody does: breathe. Eat. Drink. Sleep. And a
| few other things that are _essential to being alive_.
|
| Things that not everybody does: _EVERYTHING else_.
| K0nserv wrote:
| This is not how `npm install` works. This misunderstanding is
| so pervasive. Unless you change stuff in `package.json` `npm
| install` will not update anything, it still installs based on
| package-lock.json.
|
| Quoting from the docs:
|
| > This command installs a package and any packages that it
| depends on. If the package has a package-lock, or an npm
| shrinkwrap file, or a yarn lock file, the installation of
| dependencies will be driven by that [..]
| cxr wrote:
| What everyone should all be doing is practicing the decades-old
| discipline of source control. Attacks of the form described in
| the post, where a known-good, uncompromised dependency is
| compromised at the "supply chain" level, can be 100% mitigated--
| not fractionally or probabilistically--by cutting out the
| vulnerable supply chain. The fact that people are still dragging
| their feet on this and resist basic source control is the only
| reason why this class of attack is even possible. That vendoring
| has so many other benefits and solves other problems is even more
| reason to do so.
|
| Stacking up more sub-par tooling is not going to solve anything.
|
| Fortunately this is a problem that doesn't even have to exist,
| and isn't one that anyone falls into naturally. It's a problem
| that you have to actively opt into by taking steps like adding
| things to .gitignore to exclude them from source control,
| downloading and using third-party tools in a way that introduces
| this and other problems, et cetera--which means you can avoid all
| of it by simply not taking those extra steps.
|
| (Fun fact: on a touch-based QWERTY keyboard, the gesture to input
| "vendoring" by swiping overlaps with the gesture for
| "benefitting".)
| dbdr wrote:
| Doesn't vendoring solve the supply chain issue in the same way
| as picking a dependency version and never upgrading would?
| (especially if your package manager includes a hash of the
| dependency in a lock file)
| cxr wrote:
| Mm, no.
|
| The practice I described is neither "never upgrading" your
| dependencies nor predicated upon that. And your comment
| ignores that there's an entire other class of problem I
| alluded to (reproducibility/failure-to-reproduce) that
| normal, basic source control ("vendoring") sidesteps almost
| entirely, if not entirely in the majority of cases. That,
| too, is without addressing the fact that what you're tacitly
| treating as bedrock is not, in fact, bedrock (even if it can
| be considered the status quo within some subculture).
|
| The problem of "supply chain attacks" (which, to reiterate,
| is coded language for "not exercising source control"--or,
| "pretending to, but not actually doing it") is a problem that
| people have to _opt_ _in_ to.
|
| It takes _extra_ tooling.
|
| It takes _extra_ steps.
|
| You have to _actively do things_ to get yourself into the
| situation where you 're susceptible to it--and then it takes
| more tooling and more effort to try and get out. In contrast,
| you can protect yourself from it +and+ solve the
| reproducibility problem all by simply doing nothing[1]--
| _not_ taking the extra steps to edit .gitignore to exclude
| third-party code (and, sometimes, _first-party_ code) from
| the SCM history, _not_ employing lock files to undo the
| consequences of selectively excluding key parts of the input
| to the build process (and your app 's behavior).
|
| All these lock file schemes that language package manager
| projects have belatedly and sheepishly began to offer as a
| solution are inept attempts at doing what your base-level RCS
| already does for free (if only all the user guides for these
| package managers weren't all instructing people to subvert
| source control to begin with).
|
| Every package manager lock file format (requirements file,
| etc.) is an inferior, ad hoc, formally-specified, error-
| prone, incompatible re-implementation of half of Git.
|
| 1. <https://news.ycombinator.com/item?id=25623388>
| jhatemyjob wrote:
| That's what I do for some of my dependencies. But it's only
| for legal reasons. If licensing isn't a concern, vendoring is
| superior.
| lijok wrote:
| Vendoring hasn't been feasible since CI became free for OSS.
|
| You cannot vendor yourself out of a nuclear waste pile that is
| the modern OSS ecosystem.
| cxr wrote:
| Not sure what any of what you wrote has to do with the
| subject at hand. Your comments are vague and come across as
| non-sequiturs at best.
|
| The only thing I've claimed is that keeping dependencies
| under source control neutralizes the supply chain attacks
| that the author of the post describes.
|
| They each belong to two totally different genres of comment.
|
| If you have something concrete to say about the relationship
| between supply chain attacks and dependencies being excluded
| from source control in lieu of late-fetching them right
| at/before build time, then go for it.
| callamdelaney wrote:
| Anything to save us from not being able to apply because of a CVE
| which is only relevant if you're doing something which we don't.
| pimlottc wrote:
| This assumes that most exploits are discovered by pro-active
| third-party security vendors, instead of being noticed in
| deployed projects. Is this actually true?
| woodruffw wrote:
| > Is this actually true?
|
| I don't know, but it's the claimed truth from a lot of vendors!
| The value proposition for a lot of supply chain security
| products is a _lot_ weaker if their proactive detection isn 't
| as strong as claimed.
| nicoburns wrote:
| I would like to see a variant of this that is based on a manual
| review/audit process rather than a time-based cooldown.
|
| Something like, upgrade once there are N independent positive
| reviews AND less than M negative reviews (where you can configure
| which people are organisations you trust to audit). And of course
| you would be able to audit dependencies yourself (and make your
| review available for others).
| testplzignore wrote:
| I've been wanting something like this for years. It's simply
| impossible for millions of companies to individually review the
| thousands of updates made to their thousands of dependencies
| every day.
|
| Imagine a world where every software update has hundreds of
| signoffs from companies across the industry. That is achievable
| if we work together. For only a few minutes a day, you too can
| save a CVSS 10.0 vulnerability from going unpatched :)
| DrScientist wrote:
| The think I find most odd about the constant pressure to update
| to the most recent and implied best version is that there is some
| implicit belief that software get's uniformly better with each
| release.
|
| Bottom line those security bugs are not all from version 1.0 ,
| and when you update you may well just be swapping known bugs for
| unknown bugs.
|
| As has been said elsewhere - sure monitor published issues and
| patch if needed but don't just blindly update.
| switchbak wrote:
| I remember this used to actually be the case, but that was many
| moons ago when you'd often wait a long time between releases.
| Or maybe the quality bar was lower generally, and it was just
| easier to raise it?
|
| These days it seems most software just changes mostly around
| the margins, and doesn't necessarily get a whole lot better.
| Perhaps this is also a sign I'm using boring and very stable
| software which is mostly "done"?
| cesarb wrote:
| > Bottom line those security bugs are not all from version 1.0,
| and when you update you may well just be swapping known bugs
| for unknown bugs.
|
| One great example of that is log4shell. If you were still using
| version 1.0 (log4j 1.x), you were not vulnerable, since the bug
| was introduced in version 2.0 (log4j 2.x). There were some
| known vulnerabilities in log4j 1.x, but the most common
| configuration (logging only to a local file or to the console,
| no remote logging or other exotic stuff) was not affected by
| any of them.
| jbkgujklgui wrote:
| The infamous log4j vulnerability was actually a _feature_
| when it was introduced. It was only several years later it
| was considered a security vulnerability. Countless of other
| techs has had the same problem, for example, ActiveX. Same
| with most downgrade attacks, they were most often considered
| a good thing (better compatibility) when introduced, only
| much later were such features considered non-good.
| edoceo wrote:
| I do this with my pull-thru-proxy. When new stuff shows up, I
| know, makes it easy to identify if urgent update is needed. I
| usually lag a bit. Make a dependency update a dedicated process.
| Update a few at a time, test, etc. slower but stable.
| programmertote wrote:
| I know it's impossible in some software stack and ecosystem. But
| I live mostly in the data world, so I usually could get away from
| such issues by aggressively keeping my upstream dependency list
| lean.
|
| P.S. When I was working at Amazon, I remember that a good number
| of on-call tickets were about fixing dependencies (in most of
| them are about updating the outdated Scala Spark framework--I
| believe it was 2.1.x or older) and patching/updating OS'es in our
| clusters. What the team should have done (I mentioned this to my
| manager) is to create clusters dynamically (do not allow long-
| live clusters even if the end users prefer it that way), and
| upgrading the Spark library. Of course, we had a bunch of other
| annual and quarterly OKRs (and KPIs) to meet, so updating Spark
| got the lowest of priorities...
| jhatemyjob wrote:
| This is a decent approach, another approach is to vendor your
| dependencies and don't update them unless absolutely necessary.
| The only thing you'd need to react to is major security
| vulnerabilities.
| compumike wrote:
| There's a tradeoff and the assumption here (which I think is
| solid) is that there's more benefit from avoiding a supply chain
| attack by _blindly (by default)_ using a dependency cooldown vs.
| avoiding a zero-day by _blindly (by default)_ staying on the
| bleeding edge of new releases.
|
| It's comparing the likelihood of an update introducing a new
| vulnerability to the likelihood of it fixing a vulnerability.
|
| While the article frames this problem in terms of _deliberate,
| intentional_ supply chain attacks, I 'm sure the majority of bugs
| and vulnerabilities were never supply chain attacks: they were
| just ordinary bugs introduced unintentionally in the normal
| course of software development.
|
| On the unintentional bug/vulnerability side, I think there's a
| similar argument to be made. Maybe even SemVer can help as a
| heuristic: a patch version increment is likely safer (less likely
| to introduce new bugs/regressions/vulnerabilities) than a minor
| version increment, so a patch version increment could have a
| shorter cooldown.
|
| If I'm currently running version 2.3.4, and there's a new release
| 2.4.0, then (unless there's a feature or bugfix I need ASAP), I'm
| probably better off waiting N days, or until 2.4.1 comes out and
| fixes the new bugs introduced by 2.4.0!
| woodruffw wrote:
| Yep, that's definitely the assumption. However, I think it's
| also worth noting that zero-days, once disclosed, _do_
| typically receive advisories. Those advisories then (at least
| in Dependabot) bypass any cooldown controls, since the thinking
| is that a _known_ vulnerability is more important to remediate
| than the open-ended risk of a compromised update.
|
| > I'm sure the majority of bugs and vulnerabilities were never
| supply chain attacks: they were just ordinary bugs introduced
| unintentionally in the normal course of software development.
|
| Yes, absolutely! The _overwhelming_ majority of vulnerabilities
| stem from _normal_ accidental bug introduction -- what makes
| these kinds of dependency compromises uniquely interesting is
| how _immediately_ dangerous they are versus, say, a DoS
| somewhere in my network stack (where I 'm not even sure it
| affects me).
| mik3y wrote:
| Could a supply chain attacker simulate an advisory-
| remediating release somehow, i.e., abuse this feature to
| bypass cooldowns?
| ted_dunning wrote:
| Of course. They can simply wait to exploit their
| vulnerability. It it is well hidden, then it probably won't
| be noticed for a while and so you can wait until it is
| running on the majority of your target systems before
| exploiting it.
|
| From their point of view it is a trade-off between volume
| of vulnerable targets, management impatience and even the
| time value of money. Time to market probably wins a lot of
| arguments that it shouldn't, but that is good news for real
| people.
| hinkley wrote:
| Defaults are always assumptions. Changing them usually means
| that you have new information.
| Too wrote:
| You should also factor in that a zero-day often isn't surfaced
| to be exploitable if you are using the onion model with other
| layers that need to be penetrated together. In contrast to a
| supply chain vulnerability that is designed to actively make
| outbound connections through any means possible.
| throwawayqqq11 wrote:
| Thank you. I was scanning this thread for anyone pointing
| this out.
|
| The cooldown security scheme appears like some inverse
| "security by obscurity". Nobody could see a backdoor,
| therefor we can assume security. This scheme stands and falls
| with the assumed timelines. Once this assumption tumbles,
| picking a cooldown period becomes guess work. (Or another
| compliance box ticked.)
|
| On the other side, the assumption can very well be sound,
| maybe ~90% of future backdoors can be mitigated by it. But
| who can tell. This looks like the survivorship bias, because
| we are making decisions based on the cases we found.
| goalieca wrote:
| I'd estimate the vast majority of CVEs in third party source
| are not directly or indirectly exploitable. The CVSS scoring
| system assumes the worst case scenario the module is deployed
| in. We still have no good way to automate adjusting the score
| or even just figuring false positive.
| lambdaone wrote:
| The big problem is the Red Queen's Race nature of development
| in rapidly-evolving software ecosystems, where everyone has to
| keep pushing versions forward to deal with their dependencies'
| changes, _as well as_ any actual software developments of their
| own. Combine that with the poor design decisions found in
| rapidly-evolving ecosystems, where everyone assumes anyting can
| be fixed in the next release, and you have a recipe for
| disaster.
| OhMeadhbh wrote:
| I'm not arguing that cooldowns are a bad idea. But I will argue
| that the article presents a simplified version of user behaviour.
| One of the reasons people upgrade their dependencies is to get
| bug fixes and feature enhancements. So there may be significant
| pressure to upgrade as soon as the fix is available, cooldowns be
| damned!
|
| If you tell people that cooldowns are a type of test and that
| until the package exits the testing period, it's not "certified"
| [*] for production use, that might help with some organizations.
| Or rather, would give developers an excuse for why they didn't
| apply the tip of a dependency's dev tree to their PROD.
|
| So... not complaining about cooldowns, just suggesting some
| verbiage around them to help contextualize the suitability of
| packages in the cooldown state for use in production. There are,
| unfortunately, several mid-level managers who are under pressure
| to close Jira tickets IN THIS SPRINT and will lean on the devs to
| cut whichever corners need to be cut to make it happen.
|
| [*] for some suitable definition of the word "CERTIFIED."
| nine_k wrote:
| There is a difference between doing $ npm i
| 78 packages upgraded
|
| and upgrading just one dependency from 3.7.2_1 to 3.7.2_2,
| after carefully looking at the code of the bugfix.
|
| The cooldown approach makes the automatic upgrades of the
| former kind much safer, while allowing for the latter approach
| when (hopefully rarely) you actually need a fix ASAP.
| andrewla wrote:
| I'm not convinced that this added latency will help, especially
| if everyone uses it. It may protect you as long as nobody else
| uses a cooldown period, but once everyone uses one then the
| window between the vulnerability being introduced and it being
| detected will expand by a similar amount, because without
| exposure it is less likely to be found.
| cmckn wrote:
| I don't think it's bad advice, it really just depends on the
| project, its dependencies, and your attack surface. I so badly
| want this era of mindlessly ticking boxes to end. Security is a
| contact sport! "Best practices" won't get you to the promised
| land, you have to actually think critically about the details day
| to day.
| abalone wrote:
| Sure, but there's an obvious tradeoff: You're also delaying the
| uptake of fixes for zero-day vulnerabilities.
|
| The article does not discuss this tradeoff.
| woodruffw wrote:
| The article assumes that engineers have the technical
| wherewithal to know when they should manually upgrade their
| dependencies!
|
| Clearly I should have mentioned that Dependabot (and probably
| others) don't consider cooldown when suggesting security
| upgrades. That's documented here[1].
|
| [1]: https://docs.github.com/en/code-
| security/dependabot/working-...
| awesome_dude wrote:
| Ye Olde "Cache Invalidation" problems really
|
| Instead of updating the cache of dependencies you have
| immediately, the suggestion is to use the cooldown to wait....
|
| As you point out, this means that you have a stale cache member
| has a critical fix applied.
|
| Next week's solution - have a dependency management tool that
| alerts you when critical fixes are created upstream for
| dependencies you have
|
| Followed by - now the zero day authors are publishing their
| stuff as critical fixes...
|
| Hilarity ensues
| jimbob45 wrote:
| Isn't this more or less what the Microsoft Long-Term Support
| (LTS) versus Short-Term Support (STS) is meant to do? LTS only
| receives critical updates but eschews all experimental/beta
| features. STS gets everything for the people that couldn't care
| less if their app gets hacked (e.g. apps like calculators,
| sandboxed internal tools, etc).
|
| I know Ubuntu and others do the same but I don't know what they
| call their STS equivalent.
| Marsymars wrote:
| Depends which Microsoft products you're referring to. If you're
| talking about .NET versions, MS explicitly says "The only
| difference is the length of support."
| xg15 wrote:
| It's a good idea, but not without weak points, I think.
|
| One of the classic scammer techniques is to introduce artificial
| urgency to prevent the victim from thinking clearly about a
| proposal.
|
| I think this would be a weakness here as well: If enough projects
| adopt a "cooldown" policy, the focus of attackers would shift to
| manipulate projects into making an exception for "their"
| dependency and install it before the regular cooldown period
| elapsed.
|
| How to do that? By playing the security angle once again: An
| attacker could make a lot of noise how a new critical
| vulnerability was discovered in their project and every dependant
| should upgrade to the emergency release as quickly as possible,
| or else - with the "emergency release" then being the actually
| compromised version.
|
| I think a lot of projects would could come under pressure to
| upgrade, if the perceived vulnerability seems imminent and the
| only point for not upgrading is some generic cooldown policy.
| __MatrixMan__ wrote:
| Along those lines: If you're packaging an exploit, it's
| probably best to fix a bug while you're at it. That way people
| who want to remove their ugly workarounds will be motivated to
| violate the dependency cooldown.
| mewpmewp2 wrote:
| How would they create that noise?
| xg15 wrote:
| Depends on the level of infiltration I guess. If the attacker
| managed to get themselves into a trusted position, as with
| the XZ backdoor, they could use the official communication
| channels of the project and possibility even file a CVE.
|
| If it's "only" technical access, it would probably be harder.
| andix wrote:
| If they file a CVE, they will draw a lot of attention from
| experts to the project. Even from people who never heard
| from this package before.
| gbin wrote:
| Feels like the tragedy of the commons: I don't want to look at
| the change, I don't want to take responsibility, somebody else
| will take care or it, I just have to wait.
|
| Ok if this is an amazing advice and the entire ecosystem does
| that: just wait .... then what? We wait even more to be sure
| someone else is affected first?
|
| Every time I see people saying you need to wait to upgrade it is
| like you are accumulating tech debt: the more you wait, the more
| painful the upgrade will be, just upgrade incrementally and be
| sure you have mitigations like 0 trust or monitoring to cut early
| any weird behavior.
| tempestn wrote:
| You're not taking on any meaningful tech debt by waiting a week
| after a new version goes public to adopt it. As the OP says,
| there are services that scan popular open source tools for
| vulnerabilities as soon as they are released; even if a large
| percentage of the user base is waiting a week to update, many
| will still be caught in that period. And for various reasons
| some will still upgrade immediately.
| bongodongobob wrote:
| This is just completely wrong. If you are talking about a
| sizeable number of devices, you're not getting anything updated
| immediately even if you wanted to. You roll out to groups over
| a period of time because you don't want to break everything if
| there are unintended consequences. Your personal device? Sure
| whatever, but any fleet of devices absolutely does not get
| immediate updates across the board.
| catlifeonmars wrote:
| You're implicitly assuming that it's exposure to downstream
| consumers that causes the malicious packages to be discovered,
| but we haven't actually seen that in the last couple of major
| supply chain attacks. Instead it just buys time for the
| maintainers to undo the damage.
| andix wrote:
| Even if less consumers will notice a compromise and report it,
| it still gives additional time for security researchers to
| analyze the packages, and for maintainers to notice themselves
| they got compromised
|
| There are a lot of companies out there, that's scan packages
| and analyze them. Maintainers might notice a compromise,
| because a new release was published they didn't authorize. Or
| just during development, by getting all their bitcoin stolen ;)
| PoignardAzur wrote:
| You could do it like a brownout, and have a random per-user
| cooldown going from a day to a month.
|
| Not enough to accumulate tech debt, enough to mitigate the
| potential impact of any supply-chain vulnerability.
| morshu9001 wrote:
| The update tech debt tends not to compound or even accumulate
| at all. Usually you'd pay the same to update v1 to v2 as you
| would for v1 to v3. Maybe skipping v2 has a negative cost even.
| cesarb wrote:
| > Maybe skipping v2 has a negative cost even.
|
| I've seen this plenty of times: v1 of some library has one
| way of doing things, v2 of that library changes to a new
| incompatible way, and then v2.1 introduces a few extra
| changes to _make it easier to port from the v1 way_. If you
| wait a while, you have to do _less_ work to update than if
| you had updated immediately.
|
| One example is Python 3. After the first few Python 3.x
| releases, a few "useless" features were introduced to make it
| easier to port code from Python 2.7 (IIRC, things like
| reintroducing the u'...' syntax for unicode strings, which
| had been removed by Python 3.0 since normal '...' strings are
| now always unicode strings).
| BrenBarn wrote:
| The culture of constant updates is a cancer on the software
| world.
| ChrisMarshallNY wrote:
| I'm an advocate for safe dependency usage.
|
| I can't in good conscience, say "Don't use dependencies," which
| solves a lot of problems, but I _can_ say "Let's be careful, out
| there," to quote Michael Conrad.
|
| I strongly suspect that a lot of dependencies get picked because
| they have a sexy Website, lots of GH stars and buzz, and cool
| swag.
|
| I tend to research the few dependencies that I use. I don't
| depend lightly.
|
| I'm also fortunate to be in the position where I don't _need_ too
| many. I am quite aware that, for many stacks, there 's no choice.
| jrowen wrote:
| _A "cooldown" is exactly what it sounds like: a window of time
| between when a dependency is published and when it's considered
| suitable for use._
|
| My understanding of a cooldown, from video games, is a period of
| time after using an ability where you can't use it again. When
| you're firing a gun and it gets too hot, you have to wait while
| it cools down.
|
| I was trying to apply this to the concept in the article and it
| wasn't quite making sense. I don't think "cooldown" is really
| used when taking a pie out of the oven, for example. I would call
| this more a trial period or verification window or something?
| bauble wrote:
| You're correct; it's a misuse of the term. Better terms would
| be embargo, quarantine, or just delay.
| 333c wrote:
| The dependency has just come out of the oven, so it's too hot
| and must cool down ;)
| jrowen wrote:
| I addressed this though lol. Nobody says "my pie is on
| cooldown."
| largbae wrote:
| Wouldn't this be a great use case of any agentic AI coding
| solution? A background agent scanning code/repo/etc and making
| suggestions both for and against dependency updates?
|
| Copilot seems well placed with its GitHub integration here, it
| could review dependency suggestions, CVEs, etc and make pull
| requests.
| andix wrote:
| One reason for cooldowns is not mentioned: maintainers often
| notice by themselves they got compromised.
|
| The attacker will try to figure out when they are the least
| available: during national holidays, when they sleep, during
| conferences they attend, when they are on sick leave, personal
| time off, ...
|
| Many projects have only a few or even only a single person,
| that's going to notice. They are often from the same country
| (time zone) or even work in the same company (they might all
| attend the same conference or company retreat weekend).
| kazinator wrote:
| Cooldowns won't do anything if it takes wide deployment in order
| for the problem to be discovered.
|
| If the code just sits there for a week without anyone looking at
| it, and is then considered cooled down just due to the passage of
| time, then the cool down hasn't done anything beneficial.
|
| A form of cooldown that could would in terms of mitigating
| problems would be a gradual rollout. The idea is that the
| published change is somehow not visible to all downstreams at the
| same time.
|
| Every downstream consumer declares a delay factor. If your delay
| factor is 15 days, then you see all new change publications 15
| days later. If your delay factor is 0 days, you see everything as
| it is published, immediately. Risk-averse organizations configure
| longer delay factors.
|
| This works because the risk-takers get hit with the problem,
| which then becomes known, protecting the risk-averse from being
| affected. Bad updates are scrubbed from the queue so those who
| have not yet received them due to their delay factor wlll not see
| those updates.
| alphazard wrote:
| For some reason everyone wants to talk about all the solutions to
| supply chain attacks except designing languages to avoid them in
| the first place.
|
| Austral[0] gets this right. I'm not a user, just memeing a good
| idea when I see it.
|
| Most languages could be changed to be similarly secure. No global
| mutable state, no system calls without capabilities, no manual
| crafting of pointers. All the capabilities come as tokens or
| objects passed in to the main, and they can be given out down the
| call tree as needed. It is such an easy thing to do at the
| language level, and it doesn't require any new syntax, just a new
| parameter in main, and the removal of a few bad ideas.
|
| [0] https://austral-lang.org/
| nextaccountic wrote:
| How does Austral avoids supply chain attacks?
| alphazard wrote:
| The typical attack tries to get the victim to load a module
| which accesses information on the filesystem or in another
| module's memory, then exfiltrate that information using
| network access. Other attacks may install backdoors used for
| infiltration once the service is deployed.
|
| If the module provides some small helper function like
| `appendSmileEmoji(x: String) -> String` then it shouldn't be
| accessing the network or the filesystem, and the only memory
| it should have access to is the string passed into the
| function. The programmer already provided enough information
| to explain to the compiler that the function should not do
| these things but poor language design allows them to happen
| anyways.
|
| Components which do have access to the network should be
| vetted thoroughly because they can facilitate infiltration
| and exfiltration. Most supply chain attacks aren't introduced
| directly in a high profile project like a web framework, they
| are introduced deeper in the dependency chain in the long
| tail of transitively imported modules.
|
| In Austral (or any language using these good ideas) a small
| poorly vetted function like `appendSmileEmoji` wouldn't be
| called with access to the filesystem or network. Without
| access to any resources, the worst it can do is append a
| frowning face or a null character or something annoying. You
| can tell by its function signature above, it doesn't take a
| capability for the network or filesystem, just a String. So
| trying to ship a change that secretly accesses the filesystem
| would produce a "variable not found" error. To even get the
| new version to compile, you would have to add an argument to
| take a filesystem capability. `appendSmileEmoji(x: String,
| fs: FS) -> String`. That breaks any dependent modules because
| they have the wrong number of arguments, and looks very
| suspicious.
| rmunn wrote:
| One guy I vaguely know (only know him online, never met him
| IRL) often says "If you're using NPM you deserve what's going
| to happen to you one of these days." (He means the package
| ecosystem, so he would say the same thing about PNPM). I don't
| agree with him -- he goes way too far with his opinion -- but I
| can't deny that he's at least partly right. Two major supply-
| chain attacks in the past month (Shai-Hulud, and that tea-
| farming attack earlier this month which AFAIK doesn't yet have
| a clever name) have served to demonstrate that he definitely
| has a point in there, even if he takes it too far. NPM has a
| long way to go in terms of hardening (they've started doing so,
| thankfully), and is a very popular target.
| kachapopopow wrote:
| pick your poison:
|
| - you are vulnerable for 7 days because of a now public update
|
| - you are vulnerable for x (hours/days) because of a supply chain
| attack
|
| I think the answer is rather simple: subscribe to a vulnerability
| feed, evaluate & update. The amount of times automatic updates
| are necessary is near zero as someone who has ran libraries that
| are at times 5 to 6 years out of date exposed to the internet
| without a single event of compromise and it's not like these were
| random services, they were viewed by hundreds of thousands of
| unique addresses. There was only 3 times in the last 4 years
| where I had to perform updates due to a publically exposed
| service where these vulnerabilities affected me.
|
| Okay, the never being compromised part is a lie because of php,
| it's always PHP (monero-miner I am sure everyone is familiar
| with). The solution for that was to stop using PHP and assosiated
| software.
|
| Another one I had problems with was CveLab (GitLab if you
| couldn't tell), there has been so many critical updates pointing
| to highly exploitable CVE's that I had decided to simply migrate
| off it.
|
| In conclusion avoiding bad software is just as important as
| updates from my experience lowering the need for quick and
| automated actions.
| torginus wrote:
| I just had a thought about dependency chain attacks - has there
| been any example where instead of overtly compromising the code,
| they injected an exploitable bug?
|
| Though I guess it'd be hard to prove intent in this case.
| Jweb_Guru wrote:
| These are just backdoors as a vulnerability class. It _does_
| happen occasionally even in high quality widely used libraries,
| but it 's very rare and usually quite sophisticated (e.g.
| https://en.wikipedia.org/wiki/XZ_Utils_backdoor in 2024). A lot
| of these attacks, even when highly sophisticated, only really
| work when the attack payload can be hidden in a binary--it's
| very very hard to insert a useful backdoor into the source code
| of an actively maintained project that receives regular code
| reviews without anyone noticing (which is not to say it's
| impossible).
| chuckadams wrote:
| It's also worth noting that the xz exploit didn't even live
| in the xz source, it was added to the source by the attacker
| before uploading it to the package build farms. With most of
| the current package repositories, there's not even an
| expectation that the bundles you get are derived from a
| particular tag or commit hash, let alone a detailed chain of
| custody. I remember RPM in the old days made a big deal out
| of "pristine sources" to which patches would be applied, but
| it still has no way to prove or enforce that claim.
| nrhrjrjrjtntbt wrote:
| Didnt know this had a name i.e. cooldowns but I totally agree.
| You want tooling to generate upgrade PRs but on a safe cadence.
| For urgent patches, i.e. zero days, a security team (internal or
| external can judge)
| weinzierl wrote:
| The article assumes that the duration of the attack window is
| fixed and independent of the cooldown period. It's not. Once
| everyone waits to update the time until the vulnerability is
| found increases and the attack window will grow.
| presentation wrote:
| What if cooldowns were implemented by a package manager
| somewhat randomized, so that it's more of a gradual rollout
| instead of a fixed cooldown period?
| xmddmx wrote:
| Why not do an empirical A/B test: Set up two honeypots (or
| perhaps 2000 for statistical significance). A gets zero updates,
| B gets all updates immediately. See which ones get pwned faster.
| presentation wrote:
| If everyone uses cooldowns would the discovery time potentially
| get pushed back?
| pigbearpig wrote:
| Probably, because researchers/vendors/maintainers aren't going
| to catch everything, but you have less exposure too.
| yyyk wrote:
| This assumes the direct mode of exploitation. The indirect mode
| (where an attacker inserts a vulnerability) does not necessarily
| have the 'short time span' issue. So not a complete solution by
| any means.
|
| Also, there should a way to distinguish between security updates
| and normal updates for this. If there is, a cooldown is a useful
| idea in general for normal updates, since (presumably) the
| current version works and the new version may introduce bugs.
| acdha wrote:
| > This assumes the direct mode of exploitation. The indirect
| mode (where an attacker inserts a vulnerability) does not
| necessarily have the 'short time span' issue. So not a complete
| solution by any means.
|
| The short time span isn't just because exploits get attention:
| it's to allow the groups which do automated analysis time to
| respond. Significantly increasing the challenge level for an
| attacker to introduce a vulnerability is a meaningful
| improvement even if it doesn't prevent that class of attack
| entirely.
| bandrami wrote:
| I always remind my team that users of Debian Stable missed
| heartbleed entirely because their OpenSSL was too old. A lot of
| people and tools have the attitude that regressions never happen.
| joshdavham wrote:
| Also interesting to note that the author is involved with the uv
| python package manager, homebrew and is a member of pypa (Python
| Packaging Authority). Very impressive!
| ivh wrote:
| He mentions uv in a footnote. I will try to remember to add
| --exclude-newer $(date -I -d '2 weeks ago') the next time I use
| uv lock --upgrade.
| illusive4080 wrote:
| The Noir iPhone extension won't darken this white page because
| the author has explicitly told Noir to not darken the page. Why
| not just make a dark mode?
| unsungNovelty wrote:
| I always liked semver. Never have I liked just churning version
| numbers. It tells you exactly what a release is. Dont even need
| to look at release notes (which some doesnt have).
|
| It helps a lot with these dev cycles.
|
| Recently, I saw even EmberJS moved away from it. I still think
| they could've done it without leaving semver. Maybe I'm wrong.
| josephcsible wrote:
| Isn't this one of those things that can only work if most people
| don't do it? If everyone did it, wouldn't something like
| https://xkcd.com/989/ result?
| OakNinja wrote:
| On dependencies, a friend of mine once said "The early bird might
| get the worm but its mouse number two that get's the cheese".
| m000 wrote:
| Isn't the title kind of self-conflicting? Dependency cooldowns
| work because not everybody is using them. The more people start
| using dependency cooldowns, the less effective they become.
| macintux wrote:
| There are comments elsewhere in the discussion indicating that
| these attacks tend to be discovered by automated scanners and
| the maintainers themselves, not by customers negatively
| impacted.
| teaganga wrote:
| i had the same thing in mind, though, is not that part of the
| process until the release of available? should not the
| release already have the maintainers input?
| jappgar wrote:
| If we all did it, it wouldn't work.
| Quizzical4230 wrote:
| Could you please elaborate on this?
| alkallio wrote:
| Because nobody would test the updates.
| morshu9001 wrote:
| Heh, my stuff is always pinned to old versions just cause I don't
| want to deal with breakages, so... way ahead of you. Or behind.
|
| Also whoever keeps changing the entire API of node-redis, please
| stop.
| snowwrestler wrote:
| When I admin'ed a bunch of Wordpress and Drupal websites (open
| source CMSes), I was very quick to install security patches, and
| relatively slow to install normal updates--for this reason.
|
| It's one of the main reasons I used popular open source software:
| so I could ride the coattails of the rest of the community.
| Basically everyone else could be my beta tester.
|
| A regular update was an _input_ to the community security
| practice, so I would let it settle for a while. A security patch
| was an _output_ of the community security practice, so I would
| install ASAP, even if it meant breaking a feature temporarily.
|
| I also manually managed dependencies as commits to the main
| codebase, meaning my entire site was one deployable object from a
| single Git repo. The "modern" practice today seems to instead
| favor a minimal repo and resolving and pulling dependencies at
| deploy time. Personally I think this is a bad idea that has
| amplified the risk of supply chain attacks.
|
| And now the idea is apparently back to: give it a little while.
| Tell the automatic dependency puller to chill out and wait.
| cryptonector wrote:
| If every dependent implements a dependency cooldown, then the
| cooldown might as well not exist.
| jbkgujklgui wrote:
| Also protects against when a library developer suddenly goes
| rogue and enshittifies the library, to make a buck or two.
| coastalpuma wrote:
| I really wish homebrew supported this feature. There are a bunch
| of terminal tools I have installed through it (and wouldn't make
| sense to sandbox) that are basically one-person projects, and I
| imagine subject to supply chain compromises. I don't urgently
| need the very latest versions for any of them, as they are
| unlikely to have exploitable security vulnerabilities on my
| machine (due to not being involved with any network facing
| service).
| Seattle3503 wrote:
| > vendors are still incentivized to discover and report attacks
| quickly, but are not as incentivized to emit volumes of blogspam
| about "critical" attacks on largely underfunded open source
| ecosystems.
|
| How?
| TZubiri wrote:
| Nice. Additionally we should be using dependency limits and
| dependency moderation.
|
| Half of my job when reviewing work is telling junior devs to try
| to do something without the dependency. Usually they just learn
| about a very basic skill.
|
| "Ok so you downloaded a thing called apscheduler, what does that
| do?" "So it's like cron?" "Can you use threads, or even better
| just use a separate process?" "Cool"
|
| My (least) favourite example is the amount of juniors that
| download a dependency from a random indian dev to store and read
| credentials from .env . Just open() and read() the security
| critical file my dude.
| ElevenLathe wrote:
| Maybe the repositories should do this by default?
___________________________________________________________________
(page generated 2025-11-22 23:01 UTC)