[HN Gopher] Linux developers patch security holes faster than an...
       ___________________________________________________________________
        
       Linux developers patch security holes faster than anyone else, says
       Google
        
       Author : rustoo
       Score  : 140 points
       Date   : 2022-02-20 12:15 UTC (10 hours ago)
        
 (HTM) web link (www.zdnet.com)
 (TXT) w3m dump (www.zdnet.com)
        
       | rurban wrote:
       | The last time I saw such a comparison HP was in the lead with
       | over 200 days to fix. This time Microsoft is in the lead with 90.
       | HP doesn't seem to produce SW anymore. Or Google's project zero
       | team doesn't care about them.
        
         | fmajid wrote:
         | Microsoft has improved dramatically compared to the days of
         | Windows 95. Apple, on the other hand, is mired in complacency.
        
           | helpfulclippy wrote:
           | I'm always really disappointed every time I hear an account
           | from security researchers about how difficult it is to deal
           | with Apple. I realize that Apple cannot possibly be alone in
           | this, but there's something especially frustrating about a
           | company that aggressively denies choice to its users, cites
           | security as a major benefit of that, and then ignores
           | critical feedback about vulnerabilities.
        
             | Mikeb85 wrote:
             | If you don't admit to vulnerabilities they don't exist,
             | right?
        
               | helpfulclippy wrote:
        
               | fmajid wrote:
               | Just one example, reneging on bug bounties is mind-
               | blowingly stupid for a $3T company. Security researchers
               | will sell what they find to the likes of NSO Pegasus
               | instead of Apple. Also, the bad blood thus created makes
               | them less likely to follow "responsible disclosure" and
               | instead just disclose them as soon as discovered for
               | kudos.
        
           | mnem wrote:
           | Doesn't the data in that blog show Apple improving faster
           | than MicroSoft? Going from taking longer than to fix bugs in
           | 2019 to being faster than them now?
           | 
           | It's pretty impressive that everyone seems to have reduced
           | their time to fix bugs. Linux is doing an amazing job there.
        
         | mrweasel wrote:
         | HP spun its software out in HP Software Division and sold the
         | whole thing to everyones favorite legacy software company:
         | Micro Focus.
        
       | fastaguy88 wrote:
       | This is an example of why "average" is not a good statistic for
       | long-tailed distributions. What does it mean that Apple took 69
       | days, or Microsoft 83? Does that mean that 90% of the bugs were
       | fixed in 2 weeks, but 10% took months (or years)? Or were half
       | fixed in a month, but the remainder took 3 months or more?
       | 
       | A much more useful number would be the median. When were half the
       | bugs fixed? And it would be nice to know about the slowest 10%.
        
         | tedunangst wrote:
         | I think you want to look at cumulative time. If,
         | hypothetically, linux fixes one bug in one day for 365 days,
         | and windows fixes one bug in ten days one time, is linux 10x
         | better?
        
           | fastaguy88 wrote:
           | The problem with any of these measures is that we do not know
           | how severe the bugs are.
           | 
           | But you are certainly correct, average (or median) time does
           | not tell us how many bugs were fixed.
        
       | bombcar wrote:
       | What does "Linux developers patch a security hole"? Time until
       | the patch is merged into Linus's tree? Or more reasonably time
       | until a patched kernel is available in a given distro?
        
         | stjohnswarts wrote:
         | Your last question is impossible to answer because distros are
         | all so different.
        
           | bombcar wrote:
           | Which is the point - time until patched in the kernel source
           | tree is a relatively useless metric unless it's pretty close
           | to when those fixes ship to Ubuntu, Debian, CentOS, Amazon
           | Linux - whatever the majority of Linux is these days.
           | 
           | Otherwise a fix in Linus' tree is about as useful as a
           | committed fix to the Windows source repository.
        
             | opan wrote:
             | >Otherwise a fix in Linus' tree is about as useful as a
             | committed fix to the Windows source repository.
             | 
             | I don't see why. Users can manually update their kernels in
             | this case. I don't believe that's an option you'll get with
             | a proprietary OS.
        
               | bombcar wrote:
               | It's a bit more useful, but the breakage has to be
               | seriously impacting my _use_ of the system to roll my own
               | kernel outside of the distro 's packaging (the only one
               | I've done is manually applying a fix to a _Gentoo_ kernel
               | and even then I just throw the patch on after the normal
               | stuff).
        
       | tpmx wrote:
       | The Linux kernel developers don't release security fixes directly
       | to end users, unlike e.g. Apple and Microsoft.
       | 
       | Perhaps the Linux numbers should also include the integration/QA
       | lead time of the distros if it's meant to be a competition.
        
         | jka wrote:
         | You make a good point here -- the time-to-deployment is
         | significant as well as the time-to-resolve.
         | 
         | It's also true that once a fix is available in _any_ Linux
         | source tree, advanced users can begin to use it from that
         | moment onwards.
         | 
         | Arguably some users could develop their own workaround or
         | solution even before one is available upstream, should they
         | want to (because the code and tools to compile it are all
         | available to them).
         | 
         | (similarly it'd be valuable to analyze how long it takes for
         | the majority of end-users to upgrade across common operating
         | systems. something like this chart[1] of Firefox version
         | traffic history, but for security fix rollout)
         | 
         | [1] -
         | https://web.archive.org/web/20201220010043/https://paulcalva...
        
           | tpmx wrote:
           | Sure - there is definitely a real benefit for advanced users
           | of the Linux kernel.
           | 
           | Epecially if your organization has someone on the Linux
           | kernel security team, I guess.
        
             | bombcar wrote:
             | This can also be a downside where the fix reveals the
             | vulnerability and it begins to be exploited before distros
             | get fixes out.
             | 
             | Iirc there recently was a big one where the fix was snuck
             | into a large Linus patch and not revealed until all distros
             | had a chance to release updated kernels.
        
       | mvanbaak wrote:
       | Linux is just a kernel. Others provide a complete os i cluding
       | ways to distribute security patches. Apples and oranges.
        
       | sitting_duck wrote:
       | The *BSDs are nonexistent in the raw data [1].
       | 
       | Of course less code leads to less security holes. But 0/376? Or
       | is project zero excluding BSDs entirely?
       | 
       | [1] https://bugs.chromium.org/p/project-
       | zero/issues/list?sort=id...
        
         | tomnipotent wrote:
         | I don't believe Chrome provides support for BSD, which explains
         | why it's missing.
        
           | sitting_duck wrote:
           | How? I referenced the underlying data. They have Microsoft
           | kernel bugs. Zoom bugs. Even systemd. What am I missing? I
           | think they coincidentally happen to have their bug tracker
           | hosted on chromium's site.
        
             | [deleted]
        
       | codethief wrote:
       | > There's a lot of FUD about how Linux is being shown recently to
       | be less secure than proprietary systems. That's nonsense. But,
       | now there are hard facts from Google's Project Zero, Google's
       | security research team, showing Linux's developers do a faster
       | job of fixing security bugs than anyone else, including Google.
       | 
       | I don't see how the Linux developers' higher speed of fixing
       | security bugs would in any way substantiate the author's claim
       | that Linux is not "less secure than proprietary systems". He
       | completely leaves aside
       | 
       | - the number of security bugs per unit time,
       | 
       | - the severity of the bugs in question, and
       | 
       | - the overall system architecture and the potential impact of
       | security bugs in general. Needless to say, this is where the
       | situation on Linux looks pretty grim compared to current
       | "proprietary systems" (which has nothing to do with the fact that
       | they're proprietary).
        
         | jchw wrote:
         | In a world where bugs are certain, unpatched (because a bug is
         | known but a patch hasn't been released) e.g. iOS is nominally
         | not more secure than patched Linux, unless possibly if you are
         | a nationstate actor hoarding zerodays. The speed of patching
         | bugs is important because most targets aren't big enough to be
         | on the receiving end of a zero day. Though I do agree that this
         | doesn't really say anything about whether one system is more or
         | less secure than others from a design standpoint, it certainly
         | matters from a practical standpoint.
        
           | tedunangst wrote:
           | If you're not concerned about zero days, then why does it
           | matter how long project zero knows about the bug before it's
           | patched?
        
             | [deleted]
        
         | kodah wrote:
         | I don't think we'll ever get a total picture on the first two
         | items from Windows or MacOS, so that's difficult to compare.
         | 
         | What's the issue with Linux's architecture? Hybrid kernels
         | aren't shown to be any more secure than a monokernel to my
         | knowledge. There is something to be said that we don't make
         | kernels for device drivers, which are basically their own
         | operating systems, but that's common to all kernels.
        
         | yjftsjthsd-h wrote:
         | What architectural/impact differences does Linux have vs
         | others?
        
       | efficax wrote:
       | Tech companies are run by product managers these days and fixing
       | a bug is not a product priority if the buggy software is still
       | making money
        
         | booleandilemma wrote:
         | Then let's make it a priority.
         | 
         | We can talk about it in the next sprint, after a story has been
         | created and groomed. Then it's just a matter of someone taking
         | it off the backlog after they've gotten through their current
         | workload and scheduled a meeting with product to make sure
         | expectations are aligned.
        
         | X6S1x6Okd1st wrote:
         | this runs counter to the trend observed in the article.
        
         | marcosdumay wrote:
         | I disagree with the "these days" part. Tech companies never
         | fixed bugs. In fact, since I can remember, they have only
         | improved on that (but they still aren't any good).
        
         | mkl95 wrote:
         | This. At most companies Agile = delivering new buggy software
         | on top of old buggy software every few weeks.
        
         | brightball wrote:
         | One of many reasons I'm a fan of SAFe (Scaled Agile Framework).
         | There are 2 backlogs, one run by product and one run by
         | dev/ops/arch/sec. Over the course of a PI (about 1 quarter),
         | capacity planning usually allocated 70/30 between the two
         | backlogs.
         | 
         | This allows your tech staff to prioritize things in their 30
         | without having to make a business case for people who don't
         | intimately understand why it's important. It's beautiful.
         | 
         | They are called the Program and Solution backlogs.
         | 
         | https://www.scaledagileframework.com/program-and-solution-ba...
        
           | sdoering wrote:
           | Interesting insight. I probably never was part of a real by
           | the book SAFe environment then.
           | 
           | In all SAFe projects I took part never was there a second
           | backlog. Or if there was one (once) the reasoning was:
           | 
           | "Let'sput everything from tech in there. We will revisit it
           | once we have done the important/business-relevant stuff."
           | 
           | My take away message is that no matter what, business either
           | cares for security or it doesn't. No framework will help
           | against the priorities of product/management if they are
           | aligned against security.
        
             | brightball wrote:
             | Your description is really why it's so important. The two
             | backlogs combined with PI planning that lets developers lay
             | out the best course of action based on the priorities
             | they've been handed is core to the entire approach.
             | 
             | Developers are expected to have a lot more control in SAFe.
             | 
             | Like anything else though, it boils down to leadership
             | understanding that giving up total control is more
             | effective for the company. Usually technical leaders
             | understand that.
        
               | kodah wrote:
               | A company and managers not respecting refactor is a
               | culture problem, having two backlogs that are weighted
               | differently is just a workaround to the culture problem.
               | That culture usually isn't from managers being ignorant
               | of the value of refactor, it's that incentives _for them_
               | (bonuses, etc) are aligned against the latter backlog. If
               | you fix the incentive, all the other pieces will fall
               | into place.
        
           | andreime wrote:
           | Where I work we brag about doing SaFe but we don't actually
           | do it :))
        
           | geertj wrote:
           | > There are 2 backlogs, one run by product and one run by
           | dev/ops/arch/sec. Over the course of a PI (about 1 quarter),
           | capacity planning usually allocated 70/30 between the two
           | backlogs.
           | 
           | Does "run by" mean that dev has to justify their projects to
           | product and vice versa? Or is this a "this is our area, we'll
           | take care of it thank you" kind of thing?
           | 
           | In my team at AWS we have ~5 backlogs each planning cycle,
           | which includes areas like 'operational risk reduction,'
           | 'operational burden reduction,' and 'adoption'. What makes it
           | work in our case is that both dev and product jointly own all
           | the backlogs, and have to come to a single recommendation on
           | allocation they put forward to the service GM. Dev has to
           | defend projects they want to do in the more engineering
           | focused backlogs, while product has to do the same for the
           | feature oriented backlog. Over time, allocations have shifted
           | significantly depending on where the service is and what
           | customers are asking for. What I like about this is that it
           | forces everybody to get along and to keep their eye on what's
           | actually important (the customer). This does require that
           | both product and dev are experienced enough to see the other
           | perspective. Product will have to understand that it's not in
           | the customer's interest if dev has a super high burden, while
           | dev will have to accept that a perfect solution is usually
           | worse than a good enough one.
        
           | LoveGracePeace wrote:
           | This sounds pretty smart. It'd sound even better if it was
           | made 50/50.
        
             | brightball wrote:
             | You can agree to make it whatever balance you want. 70/30
             | is a solid recommendation to start with though.
             | 
             | I've found that in many cases, because the developers
             | understand what needs to be done so clearly in that backlog
             | that even big items go quickly.
             | 
             | The biggest thing is just to make sure that you're not in a
             | situation of 100% features where everything else is being
             | ignored until it all catches up to you. 70/30 at least
             | forces the issue.
        
           | WJW wrote:
           | My favorite part of the SAFe is the image you get right on
           | the homepage:
           | 
           | https://www.scaledagileframework.com/
           | 
           | It has a section "agile release train" which is a literal
           | train! Everybody loves trains. It also has an icon with the
           | text "Built-in quality", so you can sleep soundly knowing
           | that you will not have quality issues anymore.
           | 
           | /s of course, but that image is really something. It's so
           | vague it can be anything you want it to; a middle managers'
           | dream.
        
             | geodel wrote:
             | Well, I found this money quote on release train:
             | 
             | " _The more alignment you have, the more autonomy you can
             | grant. The one enables the other.
             | 
             | (c) Scaled Agile, Inc. Include this copyright notice with
             | the copied content.
             | 
             | Read the FAQs on how to use SAFe content and trademarks
             | here: https://www.scaledagile.com/about/about-
             | us/permissions-faq/ Explore Training at:
             | https://www.scaledagile.com/training/calendar/ _"
             | 
             | Note that I copied only the quote. But you can see this
             | quote is eminently copyrightable. Hence the copyright
             | notice.
        
               | brightball wrote:
               | I think it's include for anything copied.
        
             | marcosdumay wrote:
             | That image (and the rest of the gallery) screams "expensive
             | low quality" with all its strength. I have never seen
             | anything that uses an image like that to be worth one's
             | time.
             | 
             | And yes, that style is fashionable on some circles. I
             | usually prefer to avoid those circles.
        
             | brightball wrote:
             | Don't get me wrong, that image is dumb. I don't know why
             | they insist on promoting it so much.
             | 
             | Whenever I teach a class or tell people about SAFe I always
             | let them know 2 things:
             | 
             | 1. All of the principles are sound and amounts to a smooth
             | combination of best practices working together. Many
             | companies are doing a lot of what's involved because of
             | that.
             | 
             | 2. There are a lot of buzzwords. You can just assume you're
             | putting "lean-agile" in front of anything like they do with
             | "quantum" in movies. It's exhausting if you don't let
             | yourself chuckle. Because of it, reading the documentation
             | can be confusing if you haven't taken a class.
        
         | [deleted]
        
         | saurik wrote:
         | Yeah: Google employees routinely leave patches for bugs in the
         | PR queue because someone hasn't either update the existing
         | tests (which clearly were adding negative value if they fail on
         | the fixed code) or write new regression tests, as if "the code
         | is already known to be wrong shouldn't trump "maybe someone in
         | the future will break it again".
        
         | [deleted]
        
       ___________________________________________________________________
       (page generated 2022-02-20 23:01 UTC)