[HN Gopher] What's wrong with enterprise Linux
       ___________________________________________________________________
        
       What's wrong with enterprise Linux
        
       Author : cylo
       Score  : 164 points
       Date   : 2023-07-17 17:34 UTC (5 hours ago)
        
 (HTM) web link (unix.foo)
 (TXT) w3m dump (unix.foo)
        
       | CrampusDestrus wrote:
       | Holy.... that Oracle linux page is so friendly it almost makes me
       | puke. I can't believe anyone would fall for them
        
         | gjsman-1000 wrote:
         | The same company that just two weeks ago was sending out
         | unsolicited emails asking companies about Java licensing - a
         | friendly shakedown for using the official JRE...
         | 
         | https://news.ycombinator.com/item?id=36599118
         | 
         | Frankly, I don't get it Oracle. I would say the priorities are
         | backwards - JRE free, Oracle Linux paid would make more
         | objective sense (especially because the JRE used to be free...
         | and there's a lot of free JREs out there.)
         | 
         | Also, don't be stupid enough to download the VirtualBox
         | Extensions package at work; or your work will probably get an
         | email from a licensing agent.
        
           | count wrote:
           | I don't think there is any cross P&L strategy at play....
        
           | azlev wrote:
           | It's an huge company with mixed incentives. Not everyone is
           | align whatever their alignment is.
        
             | CrampusDestrus wrote:
             | >Yes, we know that this is Oracle, but...
             | 
             | There might be different alignments, but you simply cannot
             | write stuff like this and get away with it if any of your
             | superiors disagree.
             | 
             | That page literally admits Oracle is an asshole company
        
           | Tainnor wrote:
           | > Also, don't be stupid enough to download the VirtualBox
           | Extensions package at work; or your work will probably get an
           | email from a licensing agent.
           | 
           | Place I used to work blocked the virtualbox home page because
           | exactly this had happened.
        
         | scrlk wrote:
         | I'm reminded of Bryan Cantrill's rant about Oracle:
         | 
         | https://www.youtube.com/watch?v=-zRN7XLCRhc&t=1981s
         | 
         | > "As you know people, as you learn about things, you realize
         | that these generalizations we have are, virtually to a
         | generalization, false. Well, except for this one, as it turns
         | out. What you think of Oracle, is even truer than you think it
         | is. There has been no entity in human history with less
         | complexity or nuance to it than Oracle. And I gotta say, as
         | someone who has seen that complexity for my entire life, it's
         | very hard to get used to that idea. It's like, 'surely this is
         | more complicated!' but it's like: Wow, this is really simple!
         | This company is very straightforward, in its defense. This
         | company is about one man, his alter-ego, and what he wants to
         | inflict upon humanity -- that's it! ...Ship mediocrity, inflict
         | misery, lie our asses off, screw our customers, and make a
         | whole shitload of money. Yeah... you talk to Oracle, it's like,
         | 'no, we don't fucking make dreams happen -- we make money!'
         | ...You need to think of Larry Ellison the way you think of a
         | lawnmower. You don't anthropomorphize your lawnmower, the
         | lawnmower just mows the lawn, you stick your hand in there and
         | it'll chop it off, the end. You don't think 'oh, the lawnmower
         | hates me' -- lawnmower doesn't give a shit about you, lawnmower
         | can't hate you. Don't anthropomorphize the lawnmower. Don't
         | fall into that trap about Oracle."
        
           | bcantrill wrote:
           | So, that rant actually _predates_ Oracle v. Google. (For a
           | bit of a history of my LISA talk that you quoted, and then my
           | added color on Oracle v. Google see [0].) Oracle has shown
           | everyone over and over again who they are; it shows how deep
           | the mistrust in Red Hat has become that Oracle seems benign
           | by comparison.
           | 
           | [0] https://www.youtube.com/watch?v=Zpnncakrelk#t=15m16s
        
       | waweic wrote:
       | As a former hobbyist that is now working a lot with RHEL as a
       | sysadmin, I was really surprised to learn how little advantage
       | there actually is with buying Red Hat Support.
       | 
       | You are always limited to their opinionated decisions on what
       | your deployment should look like* (or risk losing support), but
       | at the same time, the support you actually get is next to none.
       | 
       | If I can't fix it myself, it ain't gonna be fixed.
       | 
       | At this point, I don't know what we are paying for anyways.
       | 
       | *As an example, more recent versions of RHEL only allow the use
       | of NetworkManager for permanent network configuration. In a
       | production hypervisor system, NM is completely unsuitable in my
       | opinion. It's full of footguns and that will bite us at some
       | point
        
       | gjvc wrote:
       | I've a penny that says that this quite-recently registered site,
       | having only a single article, and by an anonymous author "Cyrus",
       | is an Oracle sockpuppet / astroturf site.                   $
       | whois unix.foo                                  Domain Name:
       | unix.foo         Registry Domain ID: DF7D1B58B-FOO
       | Registrar WHOIS Server: whois.namecheap.com         Registrar
       | URL: https://www.namecheap.com/         Updated Date:
       | 2023-05-17T16:21:39Z         Creation Date: 2023-05-12T16:21:39Z
       | Registry Expiry Date: 2024-05-12T16:21:39Z         Registrar:
       | Namecheap Inc.
        
         | cylo wrote:
         | Hi. Author here. I'm not a sockpuppet and genuinely have no
         | affiliation with Oracle. It's purely my own technical opinion.
         | 
         | I only recently registered the site and set up the blog because
         | I had nowhere else to post it and did not want to use a hosted
         | platform like Medium.
         | 
         | I was watching the recent Red Hat debacle from the sidelines
         | and had some thoughts to put together about the general
         | Enterprise Linux model.
        
           | ineedasername wrote:
           | Okay, that's fair, but do you kind of see how a shiny new
           | domain name & blog with exactly one post and an anonymous
           | author saying you can trust them because they're not
           | affiliated with Oracle...
           | 
           | Can you see how that appeal for trust hasn't been built on
           | firm ground?
        
             | noizejoy wrote:
             | I didn't see a particular appeal for trust in the OP. Not
             | mentioning anything about non association would have seemed
             | weirder.
             | 
             | Sometimes it's interesting to read or hear from someone
             | with a high trust score.
             | 
             | Other times it's interesting to read or hear a line of
             | reasoning without - or even negative - trust score.
             | 
             | Only going by trust arguably is one of the fundamental
             | problems with tribalism and echo chambers.
        
               | ineedasername wrote:
               | >I didn't see a particular appeal for trust in the OP
               | 
               | It's implicit within the statement that they don't have a
               | conflict of interest. Trust is precisely what conflicts
               | of interest are about, where their presence often (maybe
               | not always) introduces the need to be more skeptical of
               | the person's motives, i.e., less trust.
        
               | [deleted]
        
             | cylo wrote:
             | Sure, I suppose. I've never been called a sockpuppet before
             | so I'm simultaneously insulted but also...honored in an odd
             | way?
             | 
             | Anyway, I'm not sure what else you'd have me do here. We
             | all have to start somewhere. I put my thoughts out there
             | under something I directly control and I'd rather people
             | focus on the central thesis of the thoughts being proposed
             | over resorting to personal attacks.
             | 
             | I'll continue posting small essays on relevant topics under
             | the domain.
        
               | ineedasername wrote:
               | I didn't mean to say you were wrong in what you chose to
               | write, but also that it wasn't wrong for someone to be
               | highly skeptical of its authenticity. Write what you want
               | and people will take is as they will, I just wanted to
               | point out that how the GP comment took it wasn't without
               | some basis.
               | 
               | As to the merits of your thesis, I'll outline my
               | understanding of it & my thoughts in response:
               | 
               | 1) You're proposing OEL as an alternative for RHEL (and
               | other such offerings)
               | 
               | 2) People might object to it based on Oracle's reputation
               | (well deserved)
               | 
               | 3) People should put that repulsion aside though
               | because...
               | 
               | 4) Oracle really is letting people use it for free so
               | user don't have to enter into a $ relationship with this
               | predatory company
               | 
               | However:
               | 
               | People choose RHEL (or any paid enterprise distro)
               | primarily because of the support contracts that guarantee
               | service. This undermines your appeal in #3 & #4 above
               | because it requires the exact thing you're saying users
               | can avoid to overcome their repulsion.
               | 
               | Also, if enough people start using this distro then I
               | have no doubt that Oracle will at some point switch
               | things up and try to monetize that userbase under threat
               | of crippling legal bills.
               | 
               | I say all of this as someone who has actually been
               | deposed by lawyers in a lawsuit with Oracle, where events
               | leading up to the lawsuit were very much a bait-and-
               | switch. I won't bog down this comment with the details
               | but if you're interested in the exact nature of that bait
               | & switch let me know and I'll comment in reply-- it may
               | change your mind (or not) about trusting Oracle not to do
               | something like that with OEL
        
       | filereaper wrote:
       | >This collection of software will remain locked at its specific
       | version throughout the lifespan of that Enterprise Linux
       | distribution release - which is often 10 years or more.
       | 
       | Is this 10 year assumption true?
       | 
       | Are people running 10 year old versions of operating systems
       | today in non-safety or non-high security aspects?
       | 
       | 10 years ago today is when the following were roughly released:
       | 
       | - Linux 3.8.
       | 
       | - RHEL 6.4 with Linux 2.6.32-358
       | 
       | - RHEL 7 was Beta with Linux 3.10.0
       | 
       | Are these still alive out there?
        
         | axus wrote:
         | They want to install security patches, and not change anything
         | else. 8 years ago was Red Hat 6.7 and 7.0. Windows Server 2012
         | R2 and 8.1 were 10 years ago.
        
         | dogline wrote:
         | Yes, these are very much alive deep in corporate data centers,
         | and we've got to work with them.
        
         | tmottabr wrote:
         | Yes it is.. I have customers that are still on RHEL 6 today and
         | are only migrating to RHEL 7 because they were forced to..
        
         | retzkek wrote:
         | Absolutely. RHEL 7 (and downstreams) has maintenance support
         | for another year, so there are many systems that administrators
         | and now looking at upgrading. Speculatively, Red Hat timed the
         | source availability change strategically to coincide with this
         | epoch and force these administrators into a tight spot and
         | hopefully get more subscriptions.
         | 
         | Should these systems have been upgraded sooner? One year has
         | usually allowed plenty of time for testing and rollout of the
         | next major release.
        
           | tmottabr wrote:
           | And after that there is still Extended Support.
           | 
           | RHEL 6 is still under extended support until next year.
        
       | lasermike026 wrote:
       | This whole model is in decline. Most companies have moved to the
       | cloud and are using kubernetes. If you're running servers a
       | datacenter then you might use enterprise linux. You are probably
       | using some kind of virtualization like ESX or Nutanix. You might
       | be using k8s there too. The age of running your own linux servers
       | is ending in a way.
        
         | 5e92cb50239222b wrote:
         | Most unicorn companies with an average half-life measured in
         | months, maybe. Most companies worldwide across all industries,
         | probably not. The pendulum will swing the other way sooner or
         | later anyway as it always does.
        
         | ineedasername wrote:
         | >You are probably using some kind of virtualization like ESX or
         | Nutanix.
         | 
         | Yes, and we spin up RHEL in them for some of our application
         | servers.
        
         | smarx007 wrote:
         | Sorry, and where do you install k8s exactly?
        
       | Kaytaro wrote:
       | > Oracle has also publicly continued to commit to keeping "...the
       | binaries and source code for that distribution publicly and
       | freely available". At this point their distribution has been
       | around for 17 years, and they've never tried to restrict access
       | to their source code in that time.
       | 
       | I'll be impressed when they open source Oracle DB and let
       | Microsoft fork it and sell support for it. It's pretty easy to
       | commit to "open source" a copy of RHEL. Especially when linux
       | isn't core to their business but a bait and switch to get their
       | customers on Oracle's platform.
        
       | darkhelmet wrote:
       | There is another problem that wasn't covered in the article. The
       | 10+ years of stability leads to behaviors and outcomes that
       | remind me of the long-lived SSL certificate problem. Updating is
       | done so infrequently that the "how?" is forgotten. As the 10 year
       | support limit approaches, most of the old team members who did it
       | last time are gone, tech debt is through the roof, few people
       | know where everything is or how to build it, and so on.
       | Enterprise Linux "stability" enables all sorts of bad behavior if
       | your company is inclined that way.
       | 
       | LetsEncrypt did us a huge favor by forcing automation vs having
       | the guy who knows how to update the SSL certs every 4.9 years and
       | left 6 months ago. I'd like to see the RHEL stability model go
       | away too and force people to complete their automation and solve
       | the problems of being able to rebuilding on demand - and actually
       | doing it.
       | 
       | (I know, most HN folk are well disciplined but there are a lot of
       | corporate cultures that are not.)
        
         | sidpatil wrote:
         | > There is another problem that wasn't covered in the article.
         | The 10+ years of stability leads to behaviors and outcomes that
         | remind me of the long-lived SSL certificate problem. Updating
         | is done so infrequently that the "how?" is forgotten. As the 10
         | year support limit approaches, most of the old team members who
         | did it last time are gone, tech debt is through the roof, few
         | people know where everything is or how to build it, and so on.
         | 
         | This is known as the _out-of-the-loop performance problem._ [1]
         | 
         | [1] https://en.wikipedia.org/wiki/Out-of-the-
         | loop_performance_pr...
        
         | picozeta wrote:
         | How is the problem of TLS certificates related to Redhat Linux
         | or Enterprise Linux? I think these are orthogonal problems.
        
           | danieldk wrote:
           | I think it was an analogy. If you don't do a thing for a long
           | time (updating SSL certificates, updating a Linux system to a
           | major new version), the knowledge of how the systems were
           | maintained/built gets lost. If you have an automated,
           | repeatable process that moves with the times, it is more
           | likely that the process is codified (either in documentation
           | or in infrastructure as data) and easy to repeat.
        
         | mfer wrote:
         | Think of all the places Linux runs. Planes, trains, and
         | automobiles. Medical equipment. So many other places. Many
         | places that don't have readily available network access. Yet,
         | many "enterprises" need support here.
         | 
         | If a medical device or train works but needs support for years
         | and years, should someone be constantly updating Linux? What
         | about the software that runs on Linux and is tested there?
         | 
         | Considering just the modern cloud environment really limits
         | where enterprise Linux runs and is useful. And, where there are
         | calls for really long support contracts.
        
           | voakbasda wrote:
           | > If a medical device or train works but needs support for
           | years and years, should someone be constantly updating Linux?
           | What about the software that runs on Linux and is tested
           | there?
           | 
           | Yes. If you have embedded software in the field and it is
           | running on hardware that has not reached its EOL, then you
           | absolutely should be fixing bugs and vulnerabilities, doubly
           | so when that hardware is attached to any kind of network, and
           | triply so when the software talks to some kind of cloud
           | services.
           | 
           | For most products, the customer should have the power decide
           | when that hardware reaches EOL. In other words, it should be
           | illegal (and severely punished) to disable or downgrade
           | devices remotely, whether by abdicating the responsibility to
           | maintain their software or by shutting down network services
           | that those devices require to operate fully.
           | 
           | At the very least, that would prevent the proliferation of
           | pervasive networking features that have no business
           | communicating with anyone except their owners.
        
             | dijit wrote:
             | > Yes. If you have embedded software in the field and it is
             | running on hardware that has not reached its EOL, then you
             | absolutely should be fixing bugs and vulnerabilities
             | 
             | Ok, I'll be the unpopular person here.
             | 
             | Some bugs, even security ones: are ok.
             | 
             | I know that a very significant amount of software is mature
             | these days, but _sometimes_ upgrading causes different
             | bugs, which are either harder to diagnose or even
             | potentially more deadly.
             | 
             | I work in gamedev though, releases are tradeoffs with which
             | bugs we accept.
             | 
             | It is very frequently the case that fixing one bug will
             | incur several other bugs, it's just a case of understanding
             | if they're worse or not.
             | 
             | For example: I don't care that my TV will have a 1/100
             | crash when launching the settings. I will just launch
             | settings again-
             | 
             | Coincidentally: having everything constantly updating and
             | internet connected is counter-intuitive. I used to spend
             | entire evenings waiting for my PS4 to install new software
             | on itself, but the PS2 (which contained many bugs!) worked
             | much more often.
        
             | pessimizer wrote:
             | > For most products, the customer should have the power
             | decide when that hardware reaches EOL. In other words, it
             | should be illegal (and severely punished) to disable or
             | downgrade devices remotely, whether by abdicating the
             | responsibility to maintain their software or by shutting
             | down network services that those devices require to operate
             | fully.
             | 
             | I mostly agree, but the _severe_ punishment for not keeping
             | software or services updated and running should be to
             | release as much of the software as you own and a full
             | hardware spec, well before you turn off the lights. People
             | have to be allowed to get out of a business.
        
             | civilitty wrote:
             | _> If you have embedded software in the field and it is
             | running on hardware that has not reached its EOL, then you
             | absolutely should be fixing bugs and vulnerabilities,
             | doubly so when that hardware is attached to any kind of
             | network, and triply so when the software talks to some kind
             | of cloud services._
             | 
             | Talk to Qualcomm and NXP and MediaTek and Broadcom and
             | STMicroelectronics and....
             | 
             | Seriously, most of this is completely out of the product
             | engineers' control. We _can 't_ update the hardware because
             | our vendors ( _ALL_ of our vendors) control the kernel
             | selection, almost never upstream, and never keep the board
             | support packages up to date. _Never._
             | 
             | IME the only exceptions to that rule are RaspberryPi (kinda
             | sorta), AMD, and Intel. I only recently managed to finally
             | extricate all my projects from Linux 2.6 which I considered
             | a minor miracle. In 2023.
        
               | fsflover wrote:
               | One more exception is NXP i.MX 8M Quad, which is used in
               | a Librem 5 phone.
        
             | ThrowawayR2 wrote:
             | Doesn't matter if the manufacturer provides updates for
             | embedded software if the customer doesn't want to install
             | those updates because they'd have to incur the cost and
             | downtime of retesting and updating their upstream software
             | to address any incompatibilities caused by those fixes.
             | It's more common than you'd think.
        
           | samstave wrote:
           | >>*Think of all the places Linux runs. """Planes, trains, and
           | automobiles""". Medical equipment.*
           | 
           | Where is your _other_ embedded equipment???
           | 
           | I love that movie.
        
         | myself248 wrote:
         | Likewise, it's been suggested that the 19.6-year (1024-week)
         | GPS epoch is pessimal. Rollover is infrequent enough to be
         | ignored, but frequent enough to actually happen and cause
         | problems.
         | 
         | Folks who know such things better than I do, have suggested
         | that it would've been far better at like a 64-week rollover (or
         | just chop it to 52 and leave part of the code space unused),
         | that way everyone would have to have a plan for it. Nobody
         | could claim they don't expect their hardware to be in use 64
         | weeks in the future therefore they can ignore rollover.
        
           | ghaff wrote:
           | Funnily enough, I had the Unix epoch time question come up
           | with a customer (who makes very long-lived pseudo-embedded
           | systems) come up in discussion last week.
        
         | jsight wrote:
         | Having worked in both kinds of cultures, I tend to agree.
         | Keeping up is ultimately less pain than trying to upgrade
         | things in huge chunks.
         | 
         | But it can be really hard to change the culture at a place that
         | has a long history of "ain't broke, don't fix it" engineering.
        
           | wkat4242 wrote:
           | Less pain yes but more efficient? I'm not sure.
           | 
           | The places that don't do constant upgrades also don't usually
           | have teams looking after that. If they time it right they can
           | do with less people.
           | 
           | Of course it's less reliable not having as much active
           | knowledge but I do think it can be cheaper if nothing goes
           | wrong.
        
             | jsight wrote:
             | Yes, that can be the tradeoff, and a reasonable one in the
             | right circumstances. Some projects are like that where
             | there is really no team dedicated to anything more than
             | keeping the lights on.
             | 
             | In my comment, I was thinking of well staffed (or at least
             | close-enough to well staffed) teams making deliberate
             | decisions to defer.
        
         | bluGill wrote:
         | Years back (late 1990s) I worked with some mainframe people.
         | They bragged that everything is hot fixable on the fly so they
         | can apply security updates or replace broken hardware without
         | rebooting. Then they admitted they schedule a reboot every 6
         | months anyway. Turns out the redundant backup power supply
         | failed in at the same time and one hot patch was not applied to
         | startup scripts and it took a week to figure out what was
         | missing so the system booted again. By rebooting every 6 months
         | they remember everything and so can get the system back up.
         | 
         | I probably have some details wrong in the story above. I worked
         | with those people, but never on the mainframe. I think the
         | point stands though, if you don't do something often it can't
         | be done.
        
           | wmf wrote:
           | This doesn't surprise me. Mainframes aren't just about never
           | failing; they have a whole culture, including ops, around
           | providing availability in ways that actually work.
        
             | pjmlp wrote:
             | Starting by having systems programming languages that
             | actually have proper strings, arrays and bounds checking.
        
               | [deleted]
        
               | reverius42 wrote:
               | What are some of those languages? I'm curious to learn
               | more.
        
               | pjmlp wrote:
               | Several PL/I dialects e.g. PL/S and PL.8, BLISS,
               | Modula-2, ESPOL/NEWP for example.
               | 
               | Also Pascal and BASIC compilers with several extensions,
               | e.g. VMS Pascal and VMS BASIC.
        
             | tjalfi wrote:
             | I've also heard of teams that shut down the mainframe for
             | an hour during a time change. It's an easy way to avoid
             | application issues for a small amount of downtime.
        
         | wkat4242 wrote:
         | In my experience updates are not forgotten. Even automated.
         | 
         | Upgrades however are a different story. The major version
         | changes require a ton of testing and manual massaging. This is
         | why enterprises like to have that infrequently. For the systems
         | that are easier you can still choose to follow the releases
         | quickly.
         | 
         | Because security patches are being back ported is usually not a
         | real issue.
        
         | 2OEH8eoCRo0 wrote:
         | I agree but I don't think that's SUSE's or Red Hat's problem.
         | If you deliver a solid and stable product humans will get
         | complacent.
        
         | thesuperbigfrog wrote:
         | >> I'd like to see the RHEL stability model go away too and
         | force people to complete their automation and solve the
         | problems of being able to rebuilding on demand - and actually
         | doing it.
         | 
         | So how could this be accomplished?
         | 
         | A Nix-OS style approach coupled with an immutable OS core?
         | 
         | It is much harder to offer stability guarantees than to just
         | publish updates in a rolling release fashion. And yet big
         | organizations pay big money for that stability or the support
         | to poke an enterprise software provider to get stuff working
         | for their needs.
        
           | sam_lowry_ wrote:
           | ArchLinux-style rather thsn NixOS style. Just roll the
           | updates when they are ready into your very own test, int,
           | acc, and finally prod.
        
             | thesuperbigfrog wrote:
             | >> ArchLinux-style rather thsn NixOS style. Just roll the
             | updates when they are ready into your very own test, int,
             | acc, and finally prod.
             | 
             | The issue is that a rolling-release approach does not have
             | stability guarantees and forces everything to be upgrading
             | all the time.
             | 
             | This does not work very well if you have specialized
             | hardware or scientific equipment. If the drivers for your
             | lab equipment work with a given release of an enterprise
             | linux, you can't just jump on the next release until you
             | have working drivers ready.
             | 
             | The same is true if you are working with some enterprise
             | software which is only certified to work with a given
             | release of an enterprise linux. Would you really want to
             | run business critical software on a version of the
             | operating system which is not (yet) supported by the
             | vendor?
        
             | AnonymousPlanet wrote:
             | All those hours hunting for the reasons why something
             | suddenly stopped working every two or three weeks need to
             | be paid. So maintenance cost for Linux servers would either
             | skyrocket or no updates would ever be done for years. There
             | are very good reasons why rolling releases in
             | infrastructure are basically a no-go.
        
           | fsflover wrote:
           | How about Qubes OS style, where everything runs in a VM?
        
             | thesuperbigfrog wrote:
             | >> How about Qubes OS style, where everything runs in a VM?
             | 
             | How does Qubes OS work with drivers for specialized
             | hardware such as scientific lab equipment?
        
               | hju22_-3 wrote:
               | Depends, I think. I remember you being able to finagle a
               | passthrough of devices, the underlying software can do
               | that with little issue and once passed through it
               | shouldn't be an issue, but I vaguely remember there being
               | some notion of that being dissuaded. Mostly because of
               | the increased attack surface, I think. Though that was a
               | few years ago now.
               | 
               | Qubes OS doesn't really solve much in regards to
               | stability and work required to update in a professional
               | setting though, I'd say.
        
               | fsflover wrote:
               | Usually devices are connected to specific VMs and the
               | drivers are installed inside them. VMs can run Lunux or
               | Windows. See this: https://www.qubes-os.org/doc/how-to-
               | use-devices/
        
           | pydry wrote:
           | It won't be. Not until comprehensive integration testing
           | comes as standard.
           | 
           | If you dont have comprehensive integration tests it's less
           | risky just to not upgrade unless you _really_ need to.
        
             | thesuperbigfrog wrote:
             | >> If you don't have comprehensive integration tests it's
             | less risky just to not upgrade unless you really need to.
             | 
             | Exactly. Most of us do not have sqlite-level
             | (https://www.sqlite.org/testing.html) testing so we just do
             | the best we can with the resources we have.
             | 
             | This tends to make most "enterprise" shops risk-averse and
             | the motto becomes "if it's not broke, don't fix it".
        
         | michaelt wrote:
         | _> I 'd like to see the RHEL stability model go away too and
         | force people to complete their automation and solve the
         | problems of being able to rebuilding on demand - and actually
         | doing it._
         | 
         | In this model, what happens when the next Python2->Python3
         | breaking change comes along?
        
           | danieldk wrote:
           | Using whatever Python your distribution needed is bad
           | practice. Own your application environment, there are plenty
           | of ways to do this, such as Nix and Docker, which make your
           | Python environments reproducible across systems.
           | 
           | Also, Enterprise Linus is one of the reasons (definitely not
           | the only) that the migration took such a long time. Too many
           | enterprise shops that stuck with Python 2 because it's the
           | lazy thing to do. The tech debt grows every year you don't
           | move with the ecosystem.
        
             | curt15 wrote:
             | >Using whatever Python your distribution needed is bad
             | practice. Own your application environment, there are
             | plenty of ways to do this, such as Nix and Docker, which
             | make your Python environments reproducible across systems.
             | 
             | How far down does "own your application environment"
             | extend? How about libc? What is the role of the underlying
             | OS?
        
               | sofixa wrote:
               | > What is the role of the underlying OS?
               | 
               | Ideally none, with scratch containers for applications
               | and the bare minimum running under your orchestrator
               | which becomes your main interface.
        
               | thesuperbigfrog wrote:
               | >> How far down does "own your application environment"
               | extend?
               | 
               | It depends on the needs of your application.
               | 
               | >> How about libc?
               | 
               | If you need to make sure the underlying libc has what you
               | need, you must either bring your own libc or have
               | sufficient feature test macros and adapters to account
               | for possible differences.
               | 
               | >> What is the role of the underlying OS?
               | 
               | It depends on what the application requires. What
               | operating system features, if any, do you require? Do you
               | have any timing or scheduling requirements that are
               | sensitive for your application? Do you need real-time
               | responsiveness?
               | 
               | How does the operating system handle failure scenarios?
               | What guarantees, if any, does it make when hardware
               | fails? Is it okay for your application to crash if a
               | portion of the computer's memory or disk borks?
        
         | otabdeveloper4 wrote:
         | > LetsEncrypt did us a huge favor by forcing automation vs
         | having the guy who knows how to update the SSL certs every 4.9
         | years and left 6 months ago.
         | 
         | Not in my experience. There's still a guy who goes around and
         | updates (manually) all the LetsEncrypt certificates every year.
        
           | outworlder wrote:
           | > Not in my experience. There's still a guy who goes around
           | and updates (manually) all the LetsEncrypt certificates every
           | year.
           | 
           | LetsEncrypt certificates don't last for one year, they only
           | last for 90 days, no exceptions. You may be thinking about
           | something different.
        
             | fragmede wrote:
             | They may be talking about the certbot software itself,
             | which does the updating of certs.
        
           | vanviegen wrote:
           | Shouldn't he be going around every couple of months?
        
           | bogeholm wrote:
           | We truly live in amazing times! We have language models that
           | sound human and internet from space, but never bothered to
           | schedule that script for updating TLS certs. Or put it in
           | version control for that matter.
           | 
           | Sounds like my org :)
        
         | derekp7 wrote:
         | Don't you end up with the same problem with the automation that
         | has been running fine for 5 years, then suddenly breaks? And
         | the person that set it up is either gone, or has no clue how
         | they did it 5 years ago.
        
           | bonzini wrote:
           | The idea is that you deploy from scratch all the
           | infrastructure every 6 months, first to testing and then to
           | production.
        
             | postmodest wrote:
             | All you damn kids work in a different industry than I do.
        
             | retbull wrote:
             | Every 6 months? That seems like a pretty long window for
             | tribal knowledge to get lost. Is 6 months arbitrary or is
             | there some reasoning behind that cadence?
        
               | noizejoy wrote:
               | Arguably, tribal knowledge and the dependence on it need
               | to be managed as much as anything else.
        
           | U2EF1 wrote:
           | LE by default successfully ran 2 months prior. 2 months and 5
           | years are two completely different worlds in terms of bit
           | rot. That and there are many generic tutorials and scripts
           | and knowledgeable devs for configuring LE fresh.
        
           | justinsaccount wrote:
           | Before LE almost no one automated SSL cert refresh. Depending
           | on your SSL cert vendor you couldn't automate things even if
           | you wanted to. It's not that the automation ran fine for five
           | years, it's that you'd be lucky if the manual process last
           | done 5 years ago was even documented.
           | 
           | SSLMate is about as old as LE, they both started around the
           | same time.
        
           | btown wrote:
           | Recently saw a (thankfully not mission-critical) old k8s
           | cluster fall down with absurd incompatibilities between node
           | versions, cluster versions, and cert-manager versions - all
           | of which only support upgrades one version at a time. Even
           | infrastructure-as-code doesn't save you if you _need_ to
           | upgrade something but don't have the time and expertise (and
           | esoteric changelog knowledge!) to reliably upgrade everything
           | else.
        
         | bogantech wrote:
         | > I'd like to see the RHEL stability model go away too and
         | force people to complete their automation and solve the
         | problems of being able to rebuilding on demand - and actually
         | doing it.
         | 
         | Whenever there's a new distribution release it invariably
         | breaks a bunch of things with the automation and you spend more
         | time massaging your playbook so it works again than it would
         | have taken to do it by hand
        
           | mananaysiempre wrote:
           | > Whenever there's a new distribution release it invariably
           | breaks a bunch of things with the automation and you spend
           | more time massaging your playbook so it works again than it
           | would have taken to do it by hand
           | 
           | The rolling-release life is that things break constantly,
           | during each week's upgrade, but only a little bit at a time
           | (and hopefully in staging). I don't know if this is better
           | for system administration, necessarily, but if you're used to
           | a stable-release dynamic of heavy discrete breakage and piles
           | of backported patches, then you might be imagining the same
           | scale of breakage every upgrade, which is isn't the
           | experience at all. So don't discard the rolling-release
           | option because of this preconception.
        
             | [deleted]
        
             | johannes1234321 wrote:
             | The difference is: If there is a weekly breakage on a
             | weekly update, delaying with it is just part of the process
             | and timed in.
             | 
             | If you only update every few years, each update becomes a
             | full project distracting from and conflicting with other
             | projects.
        
               | dralley wrote:
               | > The difference is: If there is a weekly breakage on a
               | weekly update, delaying with it is just part of the
               | process and timed in.
               | 
               | That entirely depends on your operations model. There's a
               | difference between, say, a nuclear power plant and a colo
               | web hosting shop. With the latter, sure, no problem
               | risking "minor" weekly breakage. With the former, I'd
               | much rather have scheduled, _heavily_ tested and
               | carefully monitored maintenance windows.
               | 
               | And HN tends to underestimate the number of places like
               | the former exist. Backbones of global finance and
               | telecom, industrial facilities of all kinds, etc.
        
               | freedude wrote:
               | It is the countless places that exist at some level
               | between your two extremes of the nuclear power plant and
               | the colo web host that need Enterprise Linux.
        
               | johannes1234321 wrote:
               | A nuclear power plant control software hopefully isn't
               | connected to external systems, but fully isolated.
               | 
               | And yes, upgrading that is a full blown project.
               | 
               | It is very different from a "living" software environment
               | with ongoing development processes.
        
           | geerlingguy wrote:
           | systemd threw the biggest wrench, by far, in my automation
           | workflows (this was before containers came to dominate a lot
           | of the landscape, so everything was managed with init
           | scripts). I like it now, but it also broke things quite
           | frequently in the early days, and there was a looong period
           | of time when you had to shim software to work with systemd.
           | 
           | But even still, things like snaps, the way Debian handles
           | system Python, and various little changes that have an
           | outsize effect on automated deployment do cause a good amount
           | of churn with automation.
        
             | nijave wrote:
             | Yup, enterprise Linux insulates you from unneeded change
             | (in the business context). For most companies, systemd will
             | have no impact on the bottom line vs sysvinit vs whatever.
             | 
             | However, paying an extra engineer to sort through all the
             | changes possibly will.
             | 
             | On the other hand, there's some interesting trends like
             | monokernels and minimal OS images that leverage services
             | running off-machine instead of expecting so many local
             | services removing some of the complexity/volatility (DNS,
             | SMTP, federated login)
        
         | NegativeK wrote:
         | > complete their automation
         | 
         | There are too many places where the current guard is going to
         | have to die off before they even _start_ automation. So we're
         | looking at 20-30 years.
         | 
         | LTS is the actual sane solution for these places, despite how
         | utterly insane it is.
        
         | maxclark wrote:
         | I have a customer with systems so old they weren't at risk for
         | heartbleed. He was excited about that.
        
           | geerlingguy wrote:
           | Heh, same but for the Java Log4j vulnerability. "We haven't
           | upgraded in 10 years, and it's secure from that!"
        
         | nijave wrote:
         | Long term support allows bad behavior but I think it's still
         | useful to reduce the amount of feature/breaking changes
         | happening to software.
         | 
         | Those problems can also be mitigated with mandatory environment
         | rebuilds which is trivial for a lot of setups with
         | infrastructure as code.
         | 
         | At the extreme end, you have Kubernetes/CNCF where 6 months go
         | by and you're many versions behind with a huge changelog of
         | breaking changes you have to fix first. Stable APIs and stable
         | ABIs are very useful here (which enterprise Linux provides).
        
         | gjsman-1000 wrote:
         | I wonder if there would be a market for an enterprise-grade
         | server microkernel OS. It's not the 90s anymore - Nintendo and
         | QNX are shipping tens of millions of microkernel installs every
         | year; and hardware is fast enough that choosing correctness and
         | security over speed is a valid tradeoff. Maybe if I win the
         | lottery...
        
           | vbezhenar wrote:
           | Kaspersky recently developed their own proprietary
           | microkernel OS. AFAIK they target it for IoT, but kernel is
           | kernel, probably could be used with ordinary servers as well.
           | 
           | Main issue is drivers, of course. It's hard to beat Linux. It
           | contains open source drivers and server vendors usually
           | target Linux and Windows with their driver efforts.
        
         | pavon wrote:
         | That is a good point, however I've not heard of too many cases
         | where organizations intentionally skip RHEL releases. Systems
         | that are being actively developed do regularly upgrade through
         | each RHEL release, and the 10 year support just lets them be
         | lazy about how quickly they do so. The only systems I see
         | intentionally riding out the 10+ year support are deprecated
         | systems that are already announced to sunset by the time RHEL
         | support ends.
         | 
         | The five year reign of RHEL7 was too long and did result in the
         | very issues you bring up, but the ~3 year duration of RHEL 5,6
         | & 8 was short enough to avoid problems due to attrition in
         | enterprise settings (unlike startups which have higher
         | turnover, and not counting bus factors of one - no release
         | cycle can't solve that).
         | 
         | And like others have pointed out, automation doesn't help as
         | much when moving between releases. We have everything
         | configuration controlled with kickstart and ansible and/or
         | docker, and it is great for reproducibility within a release
         | cycle, but it doesn't save much time or knowledge between
         | releases. And Ubuntu is even worse in that regard despite
         | having a shorter release cycle.
        
           | lanstin wrote:
           | In 2017 I led a (painful) project to migrate from RHEL5 to
           | Ubuntu 16. Since then, it has been pretty easy to go to 18
           | then 20, soon 22. The previous migration was in 2010 and was
           | from RedHat 6 (not RHEL) to RHEL5. These projects were for
           | projects that are "ghost deprecated" in that not much time is
           | spent in talking about them but they are critically important
           | to the business, as profit centers and cost drivers, but not
           | the new flashy stuff. So we saved $10s of millions of dollars
           | in hardware savings mostly due to the improved schedulers in
           | 4.0.x series of kernels compared to 2.6.28 kernel. The same
           | image became the basis of the containerized version that was
           | rolled out a bit later.
        
           | soneil wrote:
           | I feel somewhat called out. By time we finished migrating
           | from centos6 to centos8, centos8 was being shot in the face.
           | Talk about "fun times".
        
           | ghaff wrote:
           | >I've not heard of too many cases where organizations
           | intentionally skip RHEL releases.
           | 
           | I assume you mean major releases. Less common than minor
           | releases but, especially for air-gapped equipment, it's not
           | that unusual.
        
           | monkeywork wrote:
           | >That is a good point, however I've not heard of too many
           | cases where organizations intentionally skip RHEL releases.
           | Systems that are being actively developed do regularly
           | upgrade through each RHEL release, and the 10 year support
           | just lets them be lazy about how quickly they do so.
           | 
           | Industry specific - but finance world, we still had
           | straggling RHEL5 machines up until a year or two ago and
           | still have a bunch of RHEL6 machines and have basically NO
           | RHEL9. The vast majority of the machines are all sitting on
           | RHEL7/8.
        
           | darkhelmet wrote:
           | It's one of the things that ground Yahoo to a halt. We spent
           | years migrating from RHEL-4 to 6, then RHEL-6 to RHEL-7, and
           | by the time the projects were pretty much complete, the next
           | sunset was approaching. My cynicism comes from seeing the bad
           | things that "Enterprise Linux" enabled there.
           | 
           | Admittedly, Yahoo was an extreme case. It never solved the
           | really building problem - the culture from the early days was
           | to compile, ship and forget. Once a RHEL-6 package was pushed
           | to our dist/yinst system (packages), it would never be
           | rebuilt unless it was 1) necessary, or 2) It was time to try
           | and figure out how to build it on RHEL-7.
           | 
           | A lot of effort was spent in the later years to try and
           | address this (by burning the old tech stack to the ground),
           | but the culture was pervasive for the longest time. If
           | 10-year-RHEL didn't exist we would have been forced to
           | address the building processes.
           | 
           | If it's hard or error prone, then do it frequently until you
           | get the process nailed down.
        
             | gjvc wrote:
             | _If it 's hard or error prone, then do it frequently until
             | you get the process nailed down._
             | 
             | Major life lesson -- practice makes perfect.
        
         | rightbyte wrote:
         | If certificate renewal is automatic why do it at all ...
        
           | vbezhenar wrote:
           | To ensure that those who possess the certificate, still
           | control the domain.
           | 
           | The main issue is that certificates should really be
           | automated by every web server by default. At least for those
           | with public IP addresses. There are servers like Caddy which
           | implemented it, but it should be basic feature that just
           | works without any additional configuration.
        
         | beefield wrote:
         | Some time ago I was listening a guy talking about (operational)
         | risk management in financial industry. One of his main points
         | was that the systems in financial organizations should _not_ be
         | completely bug-free and automated. Because when something
         | eventually happens, if there is no-one who has had to fix
         | issues in the systems regularly, there is nobody around who can
         | fix the system efficiently.
         | 
         | An example of argument that belongs to a weird class of
         | arguments you at the same time want to agree and disagree.
        
       | rwmj wrote:
       | Erm, enterprise Linux systems backport major new features all the
       | time. The version number of some software means very little. This
       | article's whole premise is flawed.
        
       | azlev wrote:
       | What's not clear to me is that newer software is safer or just
       | less tested.
       | 
       | Although it's possible that newer versions are more secure, I
       | couldn't find evidence of it yet. (Links are welcome).
        
         | [deleted]
        
         | dboreham wrote:
         | Probably more of an ass-covering situation: bugs are known in
         | older software and not doing something about a known risk is
         | obviously bad. otoh not mitigating an unknown and unidentified
         | risk (bugs in new software) is ok.
        
         | wredue wrote:
         | See. I used to think similarly. Newer software must be more
         | secure, because we definitely know about all these problems and
         | how to deal with them.
         | 
         | What that doesn't consider is that the vast majority of
         | programmers simply don't care to keep up with security
         | concerns. And that's why two of the most dangerous, easy to
         | exploit, easy to stop attacks out there (SQL injection, XSS)
         | are still among the most prevalent.
         | 
         | As long as features are growing, the claims that new is more
         | secure is difficult to believe.
        
         | dylan604 wrote:
         | that's why the whole rewrite in $languageX is so annoyingly
         | frustrating when touted as fixing so many issues just because
         | it uses $languageX. How many bugs not present in original
         | version get introduce is always swept under the rug of the
         | $languageX evangelists
        
         | Pet_Ant wrote:
         | Isn't an unknown vulnerability preferrable to a known
         | vulnerabilty?
        
           | iso1631 wrote:
           | With a known vulnerability I can put in monitoring or
           | countermeasures or work out that it doesn't affect me.
           | 
           | With an unknown vulnerability, I don't have any information
           | about it, it could be affecting me right now.
        
           | jossclimb wrote:
           | There is unknown and unreported, a large amount of unreported
           | vulns are sold on the blackmarket or hoarded for exploit by
           | foreign adversaries.
        
         | jossclimb wrote:
         | less tested..the vulnerabilities have not been found yet.
        
       | im_down_w_otp wrote:
       | I could imagine an updated take on "Enterprise Distro", in order
       | to take advantage of the constantly shifting sands, being by and
       | large defined & differentiated by some fantastical gauntlet of
       | automated V&V pipelines: big piles of test suites, fuzzers, prop
       | tests, trace-based testing, etc. etc.
       | 
       | More or less setting up a rigorous de facto compliance regime and
       | making the arms race for enterprise distros be about who can
       | establish the best regime and best gauntlet automation.
        
       | tkuraku wrote:
       | I think that one of the biggest problems for my use case is
       | license management for things outside the data center. With
       | windows I can buy a pro license and have it attached to the
       | hardware. I never have to worry about tracking or managing the
       | license. With RHEL I have to make sure that the computer is
       | registered and logged in to the license correctly. There is a lot
       | of extra work for things like developer workstations or
       | instrument controllers etc. If I could just buy a license of a
       | major version of RHEL for $200-$500 and have it locked to the
       | hardware like windows that would relieve a major administrative
       | burden.
        
       | smarx007 wrote:
       | What is Oracle going to do with the userland/RPM compat with
       | RHEL? It's wonderful that they try to ship every fix from the LTS
       | kernel tree and that UEK users already don't have an expectation
       | of a 100% identical kernel. But you still need userland
       | compatibility, which will be much harder to ensure given the
       | sheer number of RPM packages out there.
        
         | freedomben wrote:
         | The Linux kernel has a pretty strict "don't break userland"
         | policy, and if it does it's a bug, so I wouldn't expect using a
         | _newer_ kernel to be a problem.
        
           | AshamedCaptain wrote:
           | Ha. The contents of /sys for example change about every other
           | release, and yes, there's obviously software which depends on
           | it.
        
           | smarx007 wrote:
           | No, I meant to ask how Oracle is going to ensure that they
           | deliver 10s of thousands RPM packages to be sufficiently
           | compatible with those that RHEL ships (following the rhelgate
           | or whatever we should call it).
        
       | opk wrote:
       | We use Oracle Linux where I work because we used to use Solaris.
       | At one point it looked suspiciously like Redhat had explicitly
       | removed support for some Oracle hardware from their kernel and
       | the UEK saved us. Besides the kernel they also offer a few other
       | extras and optional newer versions like dtrace, support for some
       | extra filesystems and newer KVM. I get the impression that the
       | fact that they're underdogs in the Linux market helps to keep
       | them honest. Much of their key staff are clearly open source
       | advocates even if Larry only cares about money. And while the
       | Oracle support is not up to the standards of the old Sun support
       | it really isn't a bad choice. And it'd be easy to switch away if
       | we ever need to.
        
         | ineedasername wrote:
         | >Redhat had explicitly removed support for some Oracle hardware
         | 
         | Sure, if you already had Oracle as a hardware vendor then the
         | cat is out of the bag, barn doors open w/ horse gone. After
         | that, using their distro isn't going to increase your attack
         | surface much, the killer is already inside the house.
        
           | noizejoy wrote:
           | Off-topic, but I have to admit enjoying the liberally mixed
           | metaphors of things leaving and entering barns, bags and
           | buildings.
           | 
           | And the music maker side of me thinks your post has the
           | beginnings of a promising blues lyric. :-)
        
       | l0b0 wrote:
       | Reminds me of CERN Linux, which was downstream of RHEL when I was
       | there. It took several years to certify all the CERN software on
       | a new release, which surely must've cost the org many more years
       | of cumulative developer productivity.
        
       | MR4D wrote:
       | I think there is something much more fundamental going on...
       | 1. Maintenance is expensive.            2. Maintenance of living
       | systems is even more so.            3. People hate spending money
       | on maintenance.            4. You reap what you sow.
       | 
       | We can't even get bridge or road maintenance done correctly in
       | the US [0]. Why should we think that IT systems would be any
       | different when we just don't have that culture?
       | 
       | This is not a _Linux_ problem - it 's a _culture_ problem (and
       | probably goes well beyond the US).
       | 
       | [0] - https://infrastructurereportcard.org/cat-item/bridges-
       | infras...
        
         | Aerbil313 wrote:
         | Yeah. Us programmers' minds keep looking for a 'solution', but
         | this is one of those solutionless paradoxes modern technology
         | brings. If technology is developing and improving, it will need
         | maintainance. If it's mature and stable, it ideally won't need
         | maintaince or there will be enough knowledgeable workers who
         | know how to do it, and their knowledge won't get obsolete in a
         | short time. Unfortunately, and fortunately, technology is
         | improving.
        
       | itomato wrote:
       | Talk to the CFOs and CTOs. They fund the hardware refresh cycles
       | and have bonuses attached to regular appeasement of shareholders,
       | the market as well as auditors and controllers.
       | 
       | The initiative begets the spend. The spend and roadmap stretches
       | 10-12 years, chassis fans and PSUs have MTBF that conveniently
       | coincide with Moore's law and the dark arts of finance, where
       | ownership is good but only to that boundary.
        
       | jacooper wrote:
       | TBH the gist I get from this article is that the way the Linux
       | project deals with security issues is a bit of a joke.
        
       | ineedasername wrote:
       | >Oracle Linux... set aside your automatic (and justified)
       | repulsion
       | 
       | I can't for two reasons:
       | 
       | 1) The only way I avoid the repulsive side of Oracle in this is
       | if I don't care about having a reliable guaranteed enterprise
       | support contract. No one is going to have more expertise in how
       | this distro is built & functions than Oracle, and that's the no-
       | go zone.
       | 
       | 2) It's not just about the current status of the project but also
       | the future and what happens if this author is successful and lots
       | of people switch over to Oracle Linux. Nothing stops them from
       | switching things up at short notice (unless I pay them for a
       | service guarantee!) and pulling the rug out from underneath the
       | whole thing. And the more that people adopt it as their distro of
       | choice, the more likely Oracle is to look around for some way to
       | monetize it at gun-point (well, crippling legal bills at least)
        
         | monkeywork wrote:
         | Lets not forget as well that as pointed out in this article OEL
         | is essentially just a play at undercutting RHEL and is the main
         | reason (imho) that RH/IBM took the steps they have over the
         | last few months is because of OEL undercutting them (I think it
         | had VERY VERY little to do with Rocky/Alma)
        
       | TheIronMark wrote:
       | If you have to run an "Enterprise Linux", you are probably
       | required by regulations or compliance to have a support agreement
       | in place. Whether it's RedHat or Oracle or something else,
       | security is going take third place to compliance and stability.
        
         | secabeen wrote:
         | Not necessarily. RHEL-clones are ubiquitous in high-performance
         | computing. Clusters and super-computers are hard to setup,
         | consist of hundreds or thousands of nodes, and are used by
         | scientists who are not programmers by trade. The RHEL-clone
         | model works well for them, as they can't expect their users to
         | keep code up to date with changes in the OS, they are used by
         | known, mostly-trusted users, and the systems they run on
         | usually stay fixed their 5-10 year lifespan. Having a single
         | supported OS for the entire lifetime of the system saves a lot
         | of work.
        
           | itomato wrote:
           | Scientific Linux? Package availability and familiarity get
           | you a long way.
        
             | secabeen wrote:
             | Scientific Linux was an RHEL-clone, and they couldn't keep
             | it going even with RH sources.
             | 
             | > In April 2019, it was announced that feature development
             | for Scientific Linux would be discontinued
        
         | dralley wrote:
         | "regulations" or "compliance" aren't the only reasons to value
         | stability. If you have a $2 billion facility or a $200 million
         | piece of industrial equipment, you're going to place value on
         | stability regardless of what the regulations say.
        
       | cbmuser wrote:
       | >>Defenders will counter that this is a necessary tradeoff - you
       | can't have both stability and being fully up-to-date on security
       | fixes. I'm not convinced that's true as there are other Linux
       | distributions that have shown you can have both. Rolling release
       | distributions like OpenSUSE Tumbleweed follow upstream much more
       | closely while still maintaining stability through thorough
       | automated testing. Additionally, distributions like Fedora Linux,
       | while not technically a rolling release, do tend to hew close to
       | upstream versions at a more regular cadence.<<
       | 
       | I'm sorry, but this paragraph proves that the author has not
       | fully understood the purpose of enterprise distributions.
       | 
       | The primary selling point is not the stability of the software,
       | but the stability of the API _and_ ABI. There is closed source
       | software such as SAP that is compiled against the ABI of RHEL and
       | SLES. And lots of expensive enterprise hardware ships Linux
       | kernel drivers in binary form only, so that a stable, i.e. never-
       | changing kernel ABI is required. Examples for such hardware is
       | the SGI UV series whose drivers come in binary form only and
       | therefore the hardware is supported on enterprise distributions
       | only.
       | 
       | Both RedHat and SUSE put a lot of effort into keeping their API
       | and ABI stable, so that enterprise customers are guaranteed that
       | no distribution update is going to break the software that
       | they're running on the hardware they're using.
       | 
       | Also, when you deploy Linux on hundreds or thousands of clients,
       | a rolling release distribution would be a pure nightmare. Given
       | the large amount of clients and applications, there will always
       | be a combination of applications and use-cases that will break
       | after a distribution package was updated to a completely new
       | upstream version.
       | 
       | Companies don't want software that changes all the time since
       | they want to control the time for an update rollout themselves.
        
         | StillBored wrote:
         | Right, and part two, is that in the case of RHEL they backport
         | features required to run on newer hardware, or enable things
         | required for newer software stacks (crypto protocols for
         | example). It just depends on whether its possible to maintain
         | API/ABI stability and support the newer feature.
         | 
         | Vs, the other LTS distros that just track the upstream LTS
         | kernel. That kernel both breaks things, as well as is "old"
         | because its only generally taking security and bug fixes.
        
       | zokier wrote:
       | > Rolling release distributions like OpenSUSE Tumbleweed follow
       | upstream much more closely while still maintaining stability
       | through thorough automated testing.
       | 
       | That is different kind of stability. No rolling release distro
       | offers API (nevermind ABI) stability the way rhel etc do, which
       | is the big selling point. You can install updates on enterprise
       | distros without needing to worry too much about your application
       | breaking because someone decided to have some fun with the API.
        
       | mbreese wrote:
       | _> The Case for Oracle Linux_
       | 
       | I did not see that twist coming.
       | 
       | Strangely, the author makes a good point. Oracle or not, sticking
       | closer to the upstream kernel is not a bad way to manage an
       | "enterprise" kernel. Maybe we've all been too willing to sit back
       | and accept how RHEL does kernels as "the way".
        
         | zokier wrote:
         | At the same time it is not that convincing point because in lot
         | of cases the value from freezing is pretty small for kernel
         | compared to other software, because kernels strong "do not
         | break userspace" attitude. Version freezing is far more
         | valuable for various other projects that do not take backwards
         | compatibility that seriously.
        
           | mbreese wrote:
           | I thought the Linux kernel had a strong "don't break
           | userspace" attitude, but it was a free for all in kernel
           | space. If you're developing kernel modules or have custom
           | hardware with drivers, I could see having a backported kernel
           | as being a major problem for support and development.
        
         | indymike wrote:
         | > Oracle or not, sticking closer to the upstream kernel is not
         | a bad way to manage an "enterprise" kernel.
         | 
         | This is probably a lot better than expecting it to be possible
         | to maintain a secure release with a level of compatibility
         | guarantee for 10+ years.
        
           | [deleted]
        
           | dralley wrote:
           | Maybe, but it's not obvious. There are plenty of novel
           | vulnerabilities in interfaces like io_uring and so forth.
           | It's not that those features are bad, I'm just saying that
           | there are tradeoffs to always getting the new stuff.
           | 
           | Maybe the compromise solution is to use newer kernels but
           | keep certain features turned off until they "bake" properly.
        
         | weare138 wrote:
         | But what stops Oracle from just pulling a Red Hat when people
         | switch over to Oracle Linux? Oracle hasn't exactly proven
         | itself to be trustworthy over the years and that's putting it
         | lightly.
         | 
         | I think we need a fully open source alternative to RHEL not
         | bound to any company. Something akin to the Debian project that
         | can serve as an upstream reference distro and repository.
        
           | MatthiasPortzel wrote:
           | Oracle Linux, Red Hat, and Debian are all "fully open
           | source". What you mean is "not maintained by a corporation."
        
           | tosihakkeri wrote:
           | > I think we need a fully open source alternative to RHEL not
           | bound to any company
           | 
           | I believe the problem is not that there wouldn't be open
           | source alternatives but that that's not what enterprise
           | wants. Enterprise wants a company behind the distribution.
        
             | noizejoy wrote:
             | But arguably, there's also a pretty large world that wants
             | enterprise type of solutions (for some use cases) without
             | being actual enterprises.
             | 
             | Or am I the only one?
        
         | vbezhenar wrote:
         | If you're using proprietary hardware drivers which target RHEL,
         | you need stable kernel.
         | 
         | Also kernel updates breaking drivers is a thing.
         | 
         | So there's definitely value with frozen kernel version. And its
         | almost the same value as frozen software. It just works and it
         | won't break after update.
        
         | gjsman-1000 wrote:
         | I actually did really like the point; but I would be very
         | interested in hearing a rebuttal or counterargument from Red
         | Hat.
         | 
         | Linus' "security problems are just bugs" approach has been
         | worrisome.
        
           | picozeta wrote:
           | > Linus' "security problems are just bugs" approach has been
           | worrisome.
           | 
           | Aren't they?
        
           | smarx007 wrote:
           | I would expect that tracking the Linux kernel master just 4
           | weeks behind is going to cause more cases where, combined
           | with enterprise software of typical quality, you'd need
           | vendor support than with the RH way of changing as little as
           | possible for as long as possible. And support is exactly what
           | Oracle sells.
           | 
           | To be clear, I only used Oracle Linux briefly while playing
           | with the free tier of Oracle Cloud.
           | 
           | Edit: sorry, but I don't get it, please break it down for me.
           | Latest UEK 7 is based off 5.15 LTS (
           | https://docs.oracle.com/en/operating-systems/uek/ ), that
           | puts them at branching off a kernel released on 2021-10-31
           | (according to https://en.wikipedia.org/wiki/Linux_kernel_vers
           | ion_history#R... ). How is that even remotely 4 weeks behind
           | master?!
           | 
           | Edit 2: my bad, I didn't read the "behind the tip of the LTS
           | tree" part carefully. So, the difference is that Oracle tries
           | to ship all LTS kernel updates, while RH tries to cherry-pick
           | critical security and bug fixes only?
        
             | bonzini wrote:
             | Red Hat doesn't change as little as possible, at least 30%
             | of the changes to Linux make it to RHEL.
             | 
             | Most of the RHEL kernel is a few months behind upstream
             | despite the old version. The next minor release of RHEL, to
             | be released in November, probably will have features up to
             | 6.3 for many subsystems and bugfixes up to 6.5 for example.
        
               | smarx007 wrote:
               | Interesting, thank you for the details. This means that
               | in some way, RHEL is ahead of Oracle Linux in terms of
               | kernels? I don't see an UEK version tracking Linux 6.1 at
               | all, AFAIK.
        
               | bonzini wrote:
               | See here for an article that describes how it's done -->
               | https://news.ycombinator.com/item?id=36763935
        
               | cylo wrote:
               | It's true that Red Hat does pull in changes from upstream
               | for several subsystems of Linux. It's genuinely a
               | frankenkernel mixed with code from 6.3, 6.1, etc.
               | 
               | But you're still beholden to what the Red Hat maintainers
               | are pulling in and focusing on. It's still not a general
               | follow upstream wholesale.
               | 
               | You can see and track what UEK is doing by looking at
               | their Github: https://github.com/oracle/linux-
               | uek/commits/uek7/u1
        
               | [deleted]
        
           | chabad360 wrote:
           | In his defence, most bugs in the kernel can be exploited, so
           | it doesn't necessarily make sense to treat a "bug" with a PoC
           | better than one without.
        
       | samstave wrote:
       | >* _What 's wrong with enterprise Linux*_
       | 
       | For me was when I was trying to incorporate Redhat, Intel and
       | Mirantis as a secure platform for on-prem-cloud as a service
       | offering - and we were setting up the environment with RH, Intel
       | and Mirantis (I was the Mirantis tech PM) - and it was a
       | nightmare of IBM-esque beurocracy.
       | 
       | it reminded me of the nightmares of installing equipment in an
       | IBM/Intel DC in the 90s...
       | 
       | Too fn many non-technical stakes in an undefined landscape and RH
       | attempting to "feel enterprise" with Intel - and it was a
       | disaster. (two FN months to get IP allocations????)
       | 
       | Yeah - I signed off on redhat way before this - but this is what
       | made me hate RHEL.
       | 
       | They got too smarmy, just as LinuxCare did...
       | 
       | But to answer the Q -- They attempted to get 'too enterprisey
       | with it' and emulate those who they were previously trying to
       | take down.
       | 
       | (want some linuxcare stories)
        
       | its-summertime wrote:
       | What's missing is an example of a case where there is a
       | potentially relevant exploit:
       | 
       | Many times, even Debian has avoided being hit by a vulnerability,
       | just due to a good wack of issues being in fresh code changes.
       | 
       | And many times, even when the vulnerability has been a long
       | existing one: RHEL is pretty focused on defense-in-depth, through
       | compiler flag choices, SELinux setups, etc.
        
       | villgax wrote:
       | The problem is that there is Enterprise Linux. You don't see a
       | consumer windows, right?
        
         | olddustytrail wrote:
         | What do you mean? Windows 11 is consumer windows. Windows
         | Server 2022 is the enterprise version.
        
         | happymellon wrote:
         | Err, you absolutely do see Windows Pro, Enterprise, Data
         | centre.
         | 
         | And there are bigger differences between the versions of
         | Windows than Linux too.
        
         | dylan604 wrote:
         | You're right, we don't see consumer windows. We see Windows
         | Home, Windows Pro, Windows Server, Windows^N-1
        
       | dangus wrote:
       | Businesses only care about security fixes as far as compliance
       | and a reasonable level risk.
       | 
       | It's more important to them type be able to tell auditors that
       | they are following a mitigation process.
       | 
       | In my case that means all I'm doing is pulling Amazon Linux
       | images and running automatic security updates.
       | 
       | Using Amazon Linux is the path of least resistance because it
       | comes with AWS tooling and it's already optimized for AWS.
       | 
       | Everything else is basically "who cares?"
        
       | arc9693 wrote:
       | How is Kernel live patching as an approach for reaching the
       | middle ground for enterprise applications? Amazon linux 2023
       | makes use of it.
       | 
       | Note how "Ksplice was the first project for live patching the
       | Linux kernel; however, ksplice was sold to Oracle and eventually
       | changed to a closed-source tool."
       | 
       | https://www.redhat.com/en/topics/linux/what-is-linux-kernel-...
        
       ___________________________________________________________________
       (page generated 2023-07-17 23:01 UTC)