[HN Gopher] Moving the Linux Kernel to Modern C
___________________________________________________________________
Moving the Linux Kernel to Modern C
Author : chmaynard
Score : 645 points
Date : 2022-02-24 20:10 UTC (1 days ago)
(HTM) web link (lwn.net)
(TXT) w3m dump (lwn.net)
| charcircuit wrote:
| >Raising the minimum GCC version to 8.x would likely be more of a
| jump than the user community would be willing to accept at this
| point.
|
| If you are using a 0 day old kernel. Why would you be using GCC
| 5.x still. 5 1 is almost 7 years old now. 8.1 is almost 4. Can't
| people just use apt / yum / rpm / whatever to upgrade to the
| latest gcc? Is that really too much to ask of the community?
| lazide wrote:
| For an unstable branch maybe, but if people need to actually
| use this for real work they will want it compiling on something
| as well understood and baked as possible - while not being
| completely obsolete.
| charcircuit wrote:
| People doing real work are using a kernel compiled with a
| more up to date GCC. From what I can tell online even RHEL is
| up to date enough by compiling its kernel with GCC 8.x.
| jahlove wrote:
| RHEL7 is still supported [0] by Red Hat through 2026, and
| is on GCC 4.8.5 [1]
|
| [0] https://access.redhat.com/product-life-
| cycles/?product=Red%2...
|
| [1] https://distrowatch.com/table.php?distribution=redhat
| charcircuit wrote:
| Correct me if I'm wrong but doesn't RHEL stick to a
| single kernel version and then backport patches?
| yjftsjthsd-h wrote:
| Officially yes, although we can argue about how much you
| can "backport" and still count. But even so - if new code
| requires a newer C version and thus newer compiler, it's
| harder to backport.
| mwcremer wrote:
| It's not just a question of what the new version fixes, there
| is also the question of what it breaks. Sometimes new versions
| trigger latent bugs no one knew about that luckily happened to
| work fine on the old tools. Finding and fixing those can be
| difficult and time-consuming.
| charcircuit wrote:
| This argument would have merit if everyone still used 5.x to
| compile Linux, but that's simply not the truth. Most people
| are using a Linux compiled by a somewhat up to date version
| of GCC.
| mhh__ wrote:
| People love making a fuss about this kind of thing.
|
| It's totally meaningless and usually dominated by the fact
| they've been asked to migrate, but it's still true
| unfortunately.
| syncsynchalt wrote:
| You'd be surprised how much downstream grief a bump like this
| can cause in an ecosystem as broad as the Linux kernel. Dealing
| with the results can be unexpectedly overwhelming.
|
| It makes sense to play it safe and do this in smaller
| incremental jumps.
| pm215 wrote:
| I get the impression that a big part of it is just a general
| choice to be very conservative about forcing newer minimum
| versions. So to raise the version bar you have to make the
| positive case for why it's worthwhile -- merely "gcc 5 is
| ancient" doesn't suffice. The only reason it moved up from 4.9
| is actual data-loss-provoking codegen bugs in 4.9 (see
| discussion in this lkml thread:
| https://lore.kernel.org/lkml/CAHk-=wjqGRXUp6KOdx-eHYEotGvY=a...
| )...
|
| I do think they could move up a bit further, but it's useful to
| be able to just build kernels with the distro compiler and
| repology thinks that for instance Debian stretch (still an LTS
| supported version) is only gcc 6.3.
| charcircuit wrote:
| Debian stretch uses the 4.19 LTS branch. A switch to a new
| GCC would happen in a new version.
| pm215 wrote:
| The distro kernel will stay on the LTS branch, certainly --
| but the upstream kernel folk like people to be able to
| compile upstream kernels and not have to stick with the
| distro ones. (I've done it myself on occasion and i'm not a
| kernel dev; if I'd had to also get hold of a new gcc it
| would have been an irritating extra step.)
| bombcar wrote:
| The Linux community is _way way way_ larger than just the sum
| of distro user bases.
|
| Even if all distros have long ago moved to GCC 8+, you still
| have other chips and systems running, and who knows what buried
| somewhere depends on GCC 5.
| charcircuit wrote:
| You can compile Linux and a user space program with different
| versions of GCC.
| Koshkin wrote:
| As well as with Tiny C (fast).
| spicybright wrote:
| You'll have to upgrade the compiler eventually though, right?
| I don't know what that process would look like, but generally
| the longer you wait the more pain you'll face later.
| lazide wrote:
| Nope, it's open source and some companies will keep it
| around until they fold in however many years (100?)
|
| If it works, why spent money 'fixing' it after all.
| spicybright wrote:
| Compiler tech and hardware will improve drastically in
| the next 100 years. Linux can either evolve with it, or
| be put in a museum while something else takes it's place.
| Likely losing a lot of the development that's been done
| in the past 30 years.
|
| It would be like running OS/8 (for the PDP-8, popular in
| 1965) on a modern machine. No time sharing for processes,
| no network stack, only a TTY for I/O... what would you
| even do with it if you could compile it?
| lazide wrote:
| We're clearly talking about very different things.
|
| If someone made a product which uses it, and people keep
| buying it, most companies aren't going to bother with any
| of those things if they don't need it.
|
| That is of course completely different from what Linux is
| doing (as in outside of that context). It may be at
| version 510, but that doesn't mean someone won't be
| running version 1.0 somewhere. And that's fine.
| simias wrote:
| I always strongly disliked the kernel's approach to macros
| (especially how they try to masquerade as everything but macros
| with their lowercase names). More than once I wasted time
| debugging code because I didn't realize that a macro was involved
| (special mention for the cases where depending on the compilation
| flags a symbol can _sometimes_ expand to a macro and sometimes
| something else).
|
| I suppose in the case of list handling there's not really a good
| alternative though. If C had lambdas we could use inline
| functions instead but that's not a thing. Still think it ought to
| have been uppercase though.
| pabs3 wrote:
| I wonder what the implications of this are for Bootstrappable
| Builds. I guess they will just need a longer GCC versions bridge
| before getting to the step of building Linux.
|
| https://bootstrappable.org/
| [deleted]
| synergy20 wrote:
| hmm I submitted the same article 10 hours ago and not got picked
| up, where is the algorithm.
| chmaynard wrote:
| Welcome to HN. This happens all the time. I added 'Linux' to
| the title, which attracted attention.
| tomcam wrote:
| This has happened to me a number of times! I think time of day
| has a heavier influence than we might think. Submitting is only
| part of the algorithm. Both vote count and velocity of votes
| are really important too.
| progbits wrote:
| From [1]:
|
| > You are not "introducing" a new macro for this, you are
| modifying the existing one such that all users of it now have the
| select_nospec() call in it.
|
| > Is that intentional? This is going to hit a _lot_ of existing
| entries that probably do not need it at all.
|
| > Why not just create list_for_each_entry_nospec()?
|
| Let's ignore whether this patch is needed or works for now - I
| don't feel competent to comment on that. But this is such a bad
| suggestion. Instead of fixing the default to be safe, and
| possibly having a _unsafe_but_fast() variant for the places where
| it makes sense they want to keep the broken version and require
| user to explicitly opt in into safety.
|
| Same as the infamous PHP mysql_real_escape_string (don't make the
| mistake of using mysql_escape_string!) [2][3] or whole host of C
| stdlib footguns like strcpy/strncpy [4].
|
| The default, easy, obvious option should be safe. The unsafe but
| faster option should be hard to use by accident and obviously
| marked as such from the name.
|
| [1] https://lwn.net/ml/linux-kernel/Yg6iCS0XZB6EtMP7@kroah.com/
| [2] https://www.php.net/manual/en/function.mysql-escape-
| string.p... [3] https://www.php.net/manual/en/function.mysql-
| real-escape-str... [4]
| https://en.cppreference.com/w/cpp/header/cstring
| [deleted]
| masklinn wrote:
| > Same as the infamous PHP mysql_real_escape_string (don't make
| the mistake of using mysql_escape_string!) [2][3]
|
| TBF that is really a straight bridge to the mysql C API, which
| is why it's also in mysqli.
|
| MySQL actually has a third one called
| mysql_real_escape_string_quote, because if the sql mode
| NO_BACKSLASH_ESCAPES is set the escaping function needs to know
| what context it's used in, and thus what quote to double up,
| and mysql_real_escape_string will fail.
| account42 wrote:
| And of course, the real answer to wether you're supposed to
| use mysql_escape_string, mysql_real_escape_string or
| mysql_real_escape_string_quote is "why the fuck are you
| pasting queries together using string processing".
| stephenr wrote:
| I get the point you're trying to make, but the blame doesn't
| really lay with php.
|
| Like most things introduced in early php, the mysql extension
| was just wrapping c functions.
|
| The concept of "real" escape comes from mysqls' C API:
| https://dev.mysql.com/doc/c-api/5.6/en/mysql-real-escape-str...
| toast0 wrote:
| > Same as the infamous PHP mysql_real_escape_string (don't make
| the mistake of using mysql_escape_string!) [2][3] or whole host
| of C stdlib footguns like strcpy/strncpy
|
| In all of those, the enhanced function takes more paramaters.
| Switching the existing name to require a new parameter will
| break existing code, and people will not want to update. Making
| the new parameter optional is possible, but IMHO is messier
| than a new function that has required parameters and then
| deprecating the old function (and eventually removing it).
| marcosdumay wrote:
| That's the deal with backwards compatibility. Even third
| parties may depend on that name, it's not safe to go and change
| it.
|
| On userspace that's the time when you deprecate an entire
| module and switch everything into a new name. The kernel does
| that once in a long while, but it's a bit harder for them.
| marcan_42 wrote:
| The kernel doesn't care about third parties though, that's
| why the kernel API/ABI is not stable. This is one big
| advantage to doing it that way: sure, you need to have a
| consistent migration plan _in flight_ to make sure you don 't
| break mainline in the process, but you can immediately remove
| the old way without having to care about third parties, once
| you're done.
| klodolph wrote:
| This is just how you work with large code bases. Updating
| 15,000 call sites is far from trivial. These changes happen in
| multiple phases, and in each phase, you still want to have a
| working kernel.
|
| The unsafe version can be deprecated and eventually removed,
| but that is down the road.
| mamcx wrote:
| You can done it, without complex tooling:
|
| 1. Have 3 versions of the thing to change: select_nospec,
| select_nospec_safe, select_nospec_unsafe.
|
| select_nospec_unsafe is _identical_ to select_nospec. Do the
| mass replace. All stay same.
|
| 2. Delete select_nospec
|
| 3. Start migrating to select_nospec_safe. You can still do
| this in small steps, because you old thing stay.
|
| 4. Where select_nospec_unsafe must be, keep it and maybe mark
| with a comment or similar to not delete it later
|
| 5. You finish and all is changed.
|
| Maybe replace select_nospec_safe to select_nospec if wanna.
| slimsag wrote:
| When there are hundreds of branches, not just feature
| branches but real long-lived development branches, and pull
| requests are managed the way they are in the Linux kernel
| (chain of trust all the way up to Linus).. "Do the mass
| replace" becomes quite non-trivial I imagine.
| aliswe wrote:
| The branches wont compile until you fix every instance
| though, so its not that risky. just a bit messy.
| mamcx wrote:
| And if the original version is not deleted in step 2 but
| after the final, then nobody break!
| mamcx wrote:
| I not arguing is effort-free, but is effort-minimal. The
| idea allows, at most, one or two major steps, the first
| and the biggest is as simple as is possible and can be
| done in one go, and have zero possibility of breaking.
|
| The only major issues is that the change is let half-way.
| But that is only possible with MAJOR undisicpline.
|
| ---
|
| The major point, to me, is that this EXCESSIVE fear of
| breakage is not good. Yes, making upgrades is pain, but
| you CAN make the pain tolerable with some planning.
|
| Refactoring is like exercise for code: Everyone dislike
| the idea of excessive, but is GOOD.
|
| And lets be honest, among all the things that could
| requiere a refactoring, this case is on the most simples
| of the simples scenario..
| eru wrote:
| Depends on who you want to put the burden on?
|
| It's relatively easy to do the mass replace in master.
| You leave the burden of the replacement in the other
| branches to people who own those branches.
|
| Set up an automated script, to keep the 'unreplaced'
| version out of master (and people can also use that
| script to check their own branches).
|
| Not sure if that counts as non-trivial, yet?
|
| There are probably some other trade-offs involved. The
| people running kernel development ain't idiots.
| tjoff wrote:
| > _Not sure if that counts as non-trivial, yet?_
|
| Assuming perfect tooling and infrastructure doesn't sound
| like it necessarily have to be a big deal.
|
| But, how far into fantasy land are we? How much effort
| would be required to get there? Is this and all similar
| problems large enough to justify it?
|
| Is adequately tooling and infrastructure even possible?
|
| Does that make proposed changes, such as moving to C11,
| order of magnitudes more difficult?
| kragen wrote:
| This strategy handles that fine as long as none of those
| branches lasts from before step 2 until the optional step
| 6. You pull from a long-lived branch, get a compile error
| because it calls select_nospec, do the replacement at
| that callsite, and commit. A headache but a manageable
| one. If you want to backport new features to old stable
| kernel versions, you can include a patch supplying
| select_nospec_unsafe as a synonym for select_nospec.
| xiphias2 wrote:
| I'm not sure how it's done with the Linux Kernel (and git
| definately makes it hard), but at Google (using Perfoce
| model) there are tools for mass renaming that try to
| guarantee that there are no regressions. Of course in Linux
| the amount of C macros make it much harder to work on the
| syntax trees, but comparision after the preprocessor step
| should be possible.
| eru wrote:
| I don't see how Perforce makes this easier than Git?
| arka2147483647 wrote:
| The point is that the change is not a simple rename:
| struct foo *iterator;
| list_for_each_entry(iterator, &foo_list, list) {
| do_something_with(iterator); }
|
| While the new version would be something like:
| list_for_each_entry_v2(iterator, &foo_list, list) {
| do_something_with(iterator); }
|
| So the 'iterator' variable may be removed, but as this is
| c, might also be reused multiple times, or come from some
| struct member, union, or whatever. So i presume something
| like this would have to be fixed case by case.
| klodolph wrote:
| You can do these kinds of changes with clang-tidy, if
| necessary. Clang-tidy is extensible and you can write
| your own rules for transforming source code.
|
| Mind you, there are not a ton of people around who have
| the time to learn how to write custom clang-tidy rules.
| It might be done for 15,000 changes that potentially fix
| kernel vulnerabilities, though.
| pitaj wrote:
| Unless something has changed in the last month, clang-
| tidy is not extensible. You can't write your own rules
| without forking the whole project.
| klodolph wrote:
| I've worked at a couple companies that had custom clang-
| tidy checks. This is well-documented enough at this
| point, but it is still a bit arcane.
|
| http://bbannier.github.io/blog/2015/05/02/Writing-a-
| basic-cl...
|
| Yes, it involves forking. Forking isn't that big a deal.
| We even had checks that were specific to internal
| libraries.
| andi999 wrote:
| At the start forking is like no deal at all. But how do
| you get new clang tidy features from the main branch
| later into your fork; I think this is the only instance
| which I would really call technical debt.
| klodolph wrote:
| "Git pull" works pretty well to bring new clang-tidy
| features downstream. If you write a custom check, it goes
| in its own set of files, so there will be no merge
| conflicts. The only thing that would break is when the
| internal clang-tidy API changes, but clang-tidy checks
| tend to be fairly short.
|
| Really... I have worked at two companies that had
| extended clang-tidy to add custom checks for their code
| base. The whole point of clang-tidy is to automate code
| fixes in other parts of your code base. When done well,
| use of clang-tide pays down technical debt _faster_
| rather than adding to it.
| plorkyeran wrote:
| Forking to add new checks only causes problems with
| pulling from upstream when APIs you are using are changed
| upstream, which happens rarely. Forking to _modify_
| existing checks would cause you a lot of pain, but the
| solution there is to copy the existing check instead.
| karlding wrote:
| You can also write your own standalone LibTooling [0]
| app, which accomplishes something similar [1].
|
| [0] https://clang.llvm.org/docs/LibTooling.html
|
| [1]
| https://clang.llvm.org/docs/LibASTMatchersTutorial.html
| jabl wrote:
| Kernel developers have previously used a tool called
| coccinelle to do these kinds of mechanical mass changes.
| It's pretty nifty.
| slaymaker1907 wrote:
| Getting code analysis/refactoring tools to understand
| huge, complicated projects like the Linux kernel is far
| from trivial.
| klodolph wrote:
| In practice it is not that hard. The refactoring tool may
| only need to tackle one translation unit at a time. You
| create a pattern which matches some usage of a macro, and
| create some logic to write out some changes to the AST.
| It can involve a surprisingly short amount of code--
| because clang-tidy has good tools for writing patterns
| and modifying ASTs!
|
| You can brows the clang-tidy source code yourself. There
| are existing checks specific to the Linux kernel in there
| already. Well, there's one check. But if you use clang-
| tidy, you'll discover that automatic refactoring of
| extremely large code bases is within reach.
|
| https://github.com/llvm/llvm-project/tree/main/clang-
| tools-e...
|
| The only question is whether you would want to spend the
| staff hours working on a clang-tidy check. Large code
| bases are exactly where the tradeoff makes sense.
| deckard1 wrote:
| What does git or Perforce have to do with refactoring? Why
| would whatever version control you're using matter?
|
| > tools for mass renaming that try to guarantee that there
| are no regressions
|
| beyond sed or whatever a fancy IDE already does? What kind
| of voodoo magic are we talking about here? Did Google solve
| the halting problem and I just haven't noticed yet...
| plantsbeans wrote:
| I believe the commenter is referring to system where you
| take a giant "git sed" and automate a rather complicated
| process: 1. break up one huge diff into a set of many
| (hundreds or thousands) patches, where each individual
| patch only touches files in a particular subsystem. 2.
| send all those patches to the appropriate system owners
| and run the appropriate tests. 3. manage the actual
| merging of successful patches, as well as communicating
| to the original author any individual patches that might
| need more attention e.g. based on patch review feedback.
|
| as you might imagine, it's probably a pretty involved
| process to touch thousands of lines of code in a
| complicated system.
| bri3d wrote:
| Disclaimer: I haven't worked at Google, but you can
| easily read their papers like
| https://dl.acm.org/doi/pdf/10.1145/2854146 and understand
| what's going on. Refactoring tooling is integrated with
| the revision control system because in a giant codebase
| with thousands of contributors, you need to split a
| refactoring patch into smaller chunks and ideally use
| semantic rather than text-based diff tooling, or the
| changes will never get reviewed or merged before they
| conflict with more changes.
|
| As for the actual renaming, yes, definitely beyond what
| sed does but probably around what the fanciest IDE does.
| Imagine a big global symbol dependency graph produced by
| the entire build toolchain all at once and cached, I
| think?
|
| Also, this isn't the halting problem unless your codebase
| allows for fully dynamic invocation :)
| worthless-trash wrote:
| Aren't all function pointers dynamic invocation ? Or is
| "fully" one of those words that i'm not grasping in
| context.
| DaiPlusPlus wrote:
| > Aren't all function pointers dynamic invocation?
|
| Given that compilers can be smart-enough to detect "non-
| dynamic" function-pointer invocation (e.g. when an
| execution-trace proves that a function-pointer parameter
| always points to the same function address), it's not
| safe to say that "all" function pointer invocations are
| dynamic.
|
| Another case to consider is when one implements
| (Smalltalk-style) OOP with message-passing: in many cases
| it's possible to build that without needing to use
| function-pointers at all.
| eru wrote:
| Also you have a type system. And even in C function
| pointers have types.
|
| So you can exclude many things that would in-principle be
| possible.
| tialaramex wrote:
| The rename has to be atomic, which is one of the things
| your revision control system gives you, unless it's like
| CVS or something worse+. So, that's a necessary part of
| the solution space.
|
| You want there to be two states: State A where the thing
| was called old_name and State B where the thing is called
| new_name. State AB where some code thinks it is called
| old_name but other code thinks it is called new_name is
| broken, and must not exist.
|
| + This might seem outrageous to modern programmers, but
| CVS thinks in terms of files, so from its point of view
| it's fine if out of sixty files you tried to commit, 48
| of them succeeded and 12 failed. Good luck fixing the
| resulting mess.
| tonyarkles wrote:
| I remember having to go backwards from SVN to CVS for a
| project a long time ago, and I was so shaken by the fact
| that CVS couldn't do atomic commits with multiple files!
| bonzini wrote:
| Yeah, my favorite model of how git operates is "rewind
| all the way back to RCS, add atomic commits and take it
| from there".
|
| SVN, baz/bzr and friends are what you get if you add
| networking before atomic commits. Git is what you get if
| you start with the proper data model and only then add
| networking.
| patmorgan23 wrote:
| There are multiple branches being actively worked on. It
| would be very easy for someone to murge in an
| unrefacatored branch and now half the code base is using
| the old call.
| eru wrote:
| You'd want something like https://bors.tech/ to avoid
| that.
|
| Bors works for git. Google has similar tooling for their
| system. (I think it's called the 'train'.)
| klodolph wrote:
| IIRC Google works the same way. The automated refactoring
| tools (Rosie?) may make it go faster, but you generally
| don't fix up 15,000 call sites in a single change... you
| break it up into smaller batches spread across the tree.
| acidbaseextract wrote:
| It's been a long time since I was there, but I thought
| there were plenty of 15,000 call site refactorings done
| in a single final CL. Not that the Linux kernel should do
| the same!
| skybrian wrote:
| When I was there (several years ago) it was rare to do
| that across many projects. You don't want to have to roll
| back everyone due to a problem that affects one project.
| Also, there's a risk that changes that are happening in
| the meantime might mean the patch doesn't apply.
| Filligree wrote:
| For a global change? These days that 'risk' is more of a
| certainty.
| staticassertion wrote:
| Linux doesn't even have consistent testing let alone the
| kind of tooling or will that Google does.
| colin_mccabe wrote:
| The Linux kernel developers have pretty good tooling for
| refactoring. For example, they have Coccinelle and
| Sparse.
|
| 1. https://en.wikipedia.org/wiki/Coccinelle_(software) 2.
| https://en.wikipedia.org/wiki/Sparse
| marcan_42 wrote:
| Linux definitely has testing. Some helpful person added
| our repo base to the kernel-test-robot list, and now I
| regularly get e-mails when something I'm working on in
| some random WIP branch broke the build on some other
| architecture (often in COMPILE_TEST mode), or some other
| random driver. Didn't even have to do anything myself.
| It's great for avoiding "this broke the build on
| $obscure_config" issues.
| interactivecode wrote:
| I thought this was one of the main advantages of typed
| languages refactoring or renaming is safe and easy. What
| gives?
| joshuamorton wrote:
| Atomically flipping a version is still difficult, as
| anything that needs to rely on the other version has to be
| fixed forward. If you instead make 5000 (or 500) small
| changes that affect 10-15 uses, you can solve the unique
| cases or roll-back and forward.
| eru wrote:
| Whether something is (statically) typed or not is more of a
| continuum than a binary.
|
| Eg C tracks whether a variable is an int or a char. Haskell
| also tracks whether a function causes side effects or not.
| More sophisticated systems also track whether a function
| has to return eventually or could run forever.
| jayd16 wrote:
| You can still have plenty of issues crop up from code
| that's not currently in master yet, or forked projects that
| take patches sparingly, etc.
|
| Typed languages let you know whats broken at compile time
| and aides in refactoring code you can see. It doesn't help
| you refactor code you can't see.
| klodolph wrote:
| That's a good question, it's not fair that you're being
| downvoted.
|
| It's the C preprocessor that causes a mess. Tooling for C
| and C++, like automatic refactoring, lags behind tooling
| for languages like Java, C#, and Go. Refactoring tools have
| to deal with macros, conditionals inside #if/#else blocks,
| and header search paths.
|
| In this case, the refactoring involves removing a variable
| from the enclosing scope of a macro invocation. The most
| likely way to automatically refactor it would be to write a
| custom check in clang-tidy.
| kibwen wrote:
| While calling C a typed language is technically correct, I
| struggle to think of any typed language that's more weakly-
| typed than C. Arguably, Python has a stronger type system
| than C.
| tapas73 wrote:
| "static vs dynamic" and "strong vs weak" are orthogonal
| coordinates.
| Koshkin wrote:
| It seems GP was talking about the latter.
| tyingq wrote:
| The C preprocessor is arguably a not type safe language
| unto itself.
| Someone1234 wrote:
| The C preprocessor is a siren song, lures people in with
| the promise of "extra performance/features/structures for
| zero runtime cost!" but murders any readability or
| rationality within your codebase.
|
| Cool concept, but once you've dealt with any legacy
| codebase that has used it extensively, it feels like an
| anti-pattern/footgun. Ultimately you just want the
| language to do those things directly and for the compiler
| to be smart enough to optimize it later.
|
| Essentially it is too clever for its own good.
| midjji wrote:
| Its not too clever for its own good, its programmers that
| are too clever for C, and desperately need basic
| features, and its the only way to get them unless you
| switch language.
|
| Its also surreally poorly designed, encouraging worse
| habits than C itself. Avoiding definition collisions and
| lack of namespaces alone make it horrific for any
| moderate sized project and up. Combine that with the near
| complete lack of static analysis tools, and its a recipe
| for disaster.
| lupire wrote:
| No one likes the preprocessor, but it's practicaly
| required if you don't want to move to C++ with template
| metaprogramming.
| badsectoracula wrote:
| I do like the preprocessor and my major pet peeve with C
| is that the preprocessor has been stagnant for ages.
| Like, why the hell can't i do something like
| #define BASE hey #define HEADERS foo bar baz
| #append HEADERS bad bal bah #push OSSUFFIX
| #ifdef WIN32 #define OSSUFFIX win32 #else
| #define OSSUFFIX unknown #endif #foreach
| H HEADERS #eval include
| "$(BASE)/$(OSSUFFIX)/$(HEADERS)" #endfor
| #pop OSSUFFIX
|
| People go to great lengths making all sorts of weird
| structures from x-macros, repeated statements, defines
| that only exist to be used by other defines, etc all to
| work around existing preprocessor limitations - and many
| of them would simply become unnecessary if the
| preprocessor could do things like variable editing, loops
| and being able to eval its own commands.
|
| Even though some stuff can be done via language features,
| it is often necessary and more flexible to work with the
| source code itself.
| midjji wrote:
| One alternative is to just write a C code generator,
| which lets you mix C and some sensible language, eg.
| python. Then use that to generate the code which is then
| sent to gcc.
| marcosdumay wrote:
| Well, C++ templates happen on the preprocessor too.
| jcelerier wrote:
| where in hell did you learn that, this is entirely false
| __d wrote:
| Well, it is _today_.
|
| When I first learned C++, using cfront in the late 80's,
| lots of the language was implemented as C pre-processor
| macros.
| jcelerier wrote:
| maybe lots of the language, but not templates. Cfront 2
| did not have templates, and Cfront 3 did have them
| without using the preprocessor.
|
| You can even check out how it was done: https://github.co
| m/seyko2/cfront-3/blob/master/src/template....
| kllrnohj wrote:
| Not really? They're part of compilation and obey all the
| type rules & syntax of the language. They're not textual
| replacements that run in a distinctly different phase
| like macros are.
| adrianN wrote:
| No they don't. Template instantiation happens during
| compilation, way later than the preprocessor.
| lupire wrote:
| "arguably" is not needed. Preprocessor destroys most of
| the semantics and even some syntax of C.
| P_I_Staker wrote:
| Yeah, I do agree. Interestingly, you could argue that
| it's fairly strongly typed; at least according to some
| definitions (and btw "type-strength" is not a rigidly
| defined term, so the definition itself is somewhat
| debatable).
|
| I don't claim to be an expert in knowing what is/isn't in
| the various standards; I just look at the build errors
| and static analysis alarms. That said, I think the
| argument is that in the vast majority of cases it's not
| actually possible to change the type of a variable; off
| the top of my head, I can only think of pointer co-
| erosion. Otherwise, if you define an unsigned int, it
| stays an unsigned int.
|
| Now strong/weak typing isn't necessarily the same thing
| as type safety. C has always seemed astonishingly bad on
| that front. It's like they try to trick novice developers
| into thinking they have a robust type system sometimes.
|
| If you define functions implicitly they will link to any
| symbol, EVEN A VARIABLE... this is bananas. I think I
| sort of understand why compilers work this way, but it
| really feels like a bug in the language. Under some
| compilers you might not even get a warning for implicit
| function, either! TI had a compiler that hid them by
| default.
|
| Enums are garbage in C. Again, you can misuse them, and
| may not even get a warning. You can pass an enum for
| color to a function that takes an enum for kittens and
| the compiler will be happy as a clam. The way they're
| often used can cause constant implicit conversions every
| time there's an assign/compare. May not be a problem for
| most positive values, but it's annoying if you're trying
| to develop standard compliant code. MISRA defines an
| "essential type system" and normal enums usage violates
| it.
|
| I'm probably forgetting a TON of deficiencies. Please add
| them or correct me where I'm wrong.
| kache_ wrote:
| lots of ways to do this. codemod, ratchets. It's a solved
| problem
| hyperpallium2 wrote:
| _First, do not break._
| Koshkin wrote:
| Second, if it ain't broke, don't fix it.
| rfoo wrote:
| Third, accept that when someone found your code is
| broken, they don't have to tell you.
| Go0the0gophers wrote:
| midjji wrote:
| You are making the rather optimistic assumption that here is a
| single definition of the macro. There are likely dozens, most
| of which share a name and randomly overwrite each other, others
| share the name but are subtly different. Macros are a truly
| terrible way to metaprogram. In my experience, if you spot one
| in a code base and can remove it, you have removed a bug you
| never knew you had, you spot one and change it, you have broken
| more things than you will ever know.
| eatbitseveryday wrote:
| > if all goes well, the shift to C11 will happen in the next
| kernel release
| stjohnswarts wrote:
| I'll be pinning that for a couple releases unless there's a
| major security bug O_O
| yjftsjthsd-h wrote:
| Just stick to a LTS branch; no reason to give up bugfixes.
| wiineeth wrote:
| Why modern C when you can move to Rust
| Koshkin wrote:
| https://thenewstack.io/rust-in-the-linux-kernel-good-enough/
|
| > _the objective was not to rewrite the kernel's 25 million
| lines of code in Rust, but rather to augment new developments
| with the more memory-safe language than the standard C normally
| used in Linux development._
| johnisgood wrote:
| When are you going to start? Contributions are welcome.
| yjftsjthsd-h wrote:
| Because we want a kernel that works today, and make
| improvements that can ship this year, not spend a decade
| rewriting working software.
| [deleted]
| the__alchemist wrote:
| I hope this isn't too much of a derail. I'm not proficient in C,
| and am translating a C module to Rust. My biggest complaint is
| there's so much DRY! Functions, variables etc are all prefixed
| with the module names, presumably due to lack of namespaces. The
| `.c` vs `.h` dichotomy is also DRY. The subset language that uses
| `#` is also a bit of a mess.
| Jtsummers wrote:
| You're using DRY oddly, but I presume you mean "there is so
| much duplication" (which is the opposite of DRY)?
|
| For the .c/.h part, it's because of use (or intended use).
| Header files are meant to be _specifications_ (though a lot of
| implementation ends up in them, annoyingly). It actually _is_
| an example of DRY thanks to C 's need to forward declare
| things. You can't call a function the compiler doesn't know
| about, for instance, so you declare (not define) it in the
| header if it's defined in an external module. If you didn't
| have the headers, you'd need to copy/paste or handwrite the
| declarations everywhere.
| the__alchemist wrote:
| You're right - I was referring to DRY violations.
|
| Why do you think there needs to be a declaration beyond the
| definition? The definition is enough information for the
| compiler, and having both is a DRY violation.
| Jtsummers wrote:
| You need the declarations because the compiler requires it,
| it's not something that I just thought up. If you don't,
| then you get two different declarations for _bar_ in the
| following. The one you intended ( _char bar(int)_ ) and the
| one you didn't ( _int bar()_ ): int foo(int
| n) { char c = bar(n); ... }
| char bar(int n) { ... }
|
| The definition is _not_ enough information for the compiler
| in a case like this, which is also essentially the case for
| any externally defined function. The compiler doesn 't know
| anything about those unless its declaration is provided,
| which is what's in the header files.
| the__alchemist wrote:
| There's one definition of `bar` there: It's a function
| that accepts an `int` as a parameter, and outputs a
| `char`.
| Jtsummers wrote:
| Did I say there were two definitions? I said there are
| two _declarations_. Try using a function with a return
| type other than _int_ before it has been defined or
| declared in C and see what happens when you compile it.
| the__alchemist wrote:
| The function is declared/defined/described/written once,
| and used once. There is no ambiguity.
|
| I don't understand your point about using a function that
| hasn't been declared. Of course it won't work!
|
| I understand your point about using header files as
| interfaces to third-party code, but there are ways of
| doing that that don't involve duplicating all functions,
| structs etc.
| Jtsummers wrote:
| The function is used once, and because it occurs before
| the actual definition (and there was no forward
| declaration) an implicit declaration occurs (the
| erroneous _int bar()_ ). When the function definition is
| later reached by the compiler, its signature doesn't
| match the implicit declaration's signature, causing the
| problem. If the function signature matches the implicit
| declaration, then there is no problem:
| int main(int argc, char* argv[]) { printf("%d\n",
| foo(10)); return 0; } int foo(int n)
| { return n - 1; }
|
| Will work just fine, giving you only a warning but will
| print out "9" as expected.
| the__alchemist wrote:
| Yikes! The implicit declaration, if used before (as in a
| lower line number than the function definition?) sounds
| like a trap, and at the root of this.
| ylyn wrote:
| Yes.
|
| But that's C. You can't fix it without fundamentally
| altering the language.
| lukas099 wrote:
| > an implicit declaration occurs (the erroneous int bar()
|
| I still don't understand; why does the implicit
| declaration assume _int bar()_ is the signature when
| _bar_ is being passed an int and returning a value to a
| char?
| Jtsummers wrote:
| Because C doesn't do type inference.
| Gibbon1 wrote:
| Header files are a historical legacy from when C was
| developed. Because machines were small compilers/languages
| were designed to do single pass compilation on modules
| which were then linked.
|
| Friends of mine who are actual CS majors say with C it's
| not fixable without changes to the language. With Java or
| C# the compiler can do a pass and extract the object
| definitions. But because of ambiguous syntax and majorly
| because of the preprocessor it's not possible with C as it
| stands.
| P_I_Staker wrote:
| You can call a function the compiler doesn't know about, but
| god help you if there happens to be an identical symbol in
| the table; it will link to literally anything, including an
| integer. This should pretty much always be turned off by
| setting implicit functions to be errors.
|
| Also, this explains all the prefixing that OP was complaining
| about lol, or at least that's one reason. I saw a
| particularly bad bug, due to poor naming decisions. Less is
| not always more, even if it annoys you.
| quelsolaar wrote:
| In may ways you cant avoid more modern C because more modern
| standards make thing that were ambiguous more clear, and then
| compilers follow that even if you select an older standard.
|
| I think C99 introduces some bad things that should be keep out of
| any code base. Variable length arrays being the biggest one.
| Generics is another one. Being able declare variables anywhere
| doesn't make the code better but it makes styles diverge and
| that's a problem.
|
| If Linux wants to adopt C99 they should define what parts of c99
| should be adopted in their style guide.
| nybble41 wrote:
| > Being able declare variables anywhere doesn't make the code
| better...
|
| It does, though. The ability to declare variables "anywhere"
| and allows a reduction in the portion of the code where a
| variable exists uninitialized, or initialized with a dummy
| value which should never be used, or holding an obsolete value
| which is not intended to be used. It also allows more variables
| to be declared `const` (which requires them to be initialized
| in the declaration, which must occur _after_ the initial value
| is computed) which helps to detect accidental assignment and
| also makes the code easier to review.
|
| It's worth noting that you can more-or-less declare variables
| "anywhere" in C89 too--you just have to wrap the variable's
| scope in a compound statement. From that point of view, C99
| doesn't change where variables can be declared. It just lets
| you remove some syntactic clutter.
| jstimpfle wrote:
| I for one have read a lot of code with C89 declare-first style
| that was subjectively a lot less clear. This applies in
| particular to assign-once variables that depend on other
| calculations.
|
| stuff like this: [0]. I mean, what's the point of declaring
| variables like _cp_ or _err_ at the beginning when they 're
| assigned first only halfway down? What is the point of having
| _path_ around for the whole function, when it is a super
| temporary assign-once variable with a scope of 3 lines? I 'm
| having a really hard time coming up with a justification for
| this, other than "this is the way we've always done it and we
| like it" (you can find actual rants in defense of this style
| from about 10 years ago).
|
| For-loop declarations are another instance of this that make
| the code more fluent IMO. And they help reducing the scope of
| variables to a strictly smaller block. I would find it hard to
| argue that this does not remove some bugs or improve
| readability.
|
| The only thing that declare-first has going for it is that the
| variables that are in use can be seen at once, a little bit
| like in a struct declaration. Looking at how optimizers butcher
| those variables depending on liveness makes this feature seem
| less valuable, though.
|
| [0]
| https://github.com/torvalds/linux/blob/2729cfdcfa1cc49bef5a9...
| pjmlp wrote:
| That is something I fully agree with.
| JoshTriplett wrote:
| > Variable length arrays being the biggest one.
|
| The kernel already used VLAs as an extension, and is currently
| in the process of removing them from all the code using them.
| pjmlp wrote:
| Someone did a big mistake since 2018 then,
|
| https://www.phoronix.com/scan.php?page=news_item&px=Linux-
| Ki...
| LAC-Tech wrote:
| They're discussing the move to C99. Not sure that counts as
| modern C at this point.
| clhodapp wrote:
| Given that C99 is much closer in vintage to their current C89
| than it is to today's date and also just straight up really
| old, it does seem inappropriate to distinguish that as "modern
| C".
|
| That said they do seem to be discussing both C99 and C11 so it
| does seem like going with something that _would_ qualify as
| modern (especially in light of Linux 's need to be
| conservative) is actually on the table.
| dahfizz wrote:
| You also have to look at the contents of the revisions. C99
| was a massive update for C, and C11 was tiny in comparison
| (and C17 was basically nothing at all).
|
| Writing C89, without variable declaration in a for loop, and
| without // comments, etc, feels ancient. C99 on the other
| hand is what defines how C looks today.
|
| This is also why the attitude is: if they jump to c99, they
| may as well jump to c11 because they are basically the same.
| flyingfences wrote:
| They're discussing a move to C11, which counts in my book.
| tombert wrote:
| I haven't touched C in any serious capacity in quite awhile;
| how often does C get new revisions, because isn't C11 still
| more than a decade old?
|
| I understand why the linux kernel folks aren't moving to the
| bleeding edge (Linux is old, you have to do these ports
| incrementally), but I'm curious why the pace of language is
| so slow. I guess it's because C has more or less stabilized
| and thus further revisions are less necessary?
| LukeShu wrote:
| > how often does C get new revisions
|
| 1989, 1995, 1999, 2011, 2017.
|
| The 1995 one was an "amendment", not a full revision (and
| is mostly additions to libc, which Linux doesn't get to
| use; the only language change is the addition of digraphs).
|
| C17 was a "bugfix" revision; compilers will have applied
| these fixes to their C11 implementations, so in effect the
| only difference between C11 and C17 is the value of
| __STDC_VERSION__. So for most intents and purposes C11 is
| still the most recent revision.
| tombert wrote:
| Fair enough! As I said, I suppose C doesn't need to
| change that much; people use C _because_ they know what
| they 're getting, and for that to be the case, it needs
| to be stable.
| rurban wrote:
| Next are C23 (2023) and C26 (2026). C23 is already
| closed, so fixes will have to go to C26.
| dahfizz wrote:
| > but I'm curious why the pace of language is so slow. I
| guess it's because C has more or less stabilized and thus
| further revisions are less necessary?
|
| That is a big part of it. C is an old, stable, complete
| language.
|
| There is also the existence of gnu extensions, which bridge
| the gap between language revisions. For example, anonymous
| unions were added in C11 but they have been a gnu extension
| since forever.
| syncsynchalt wrote:
| > I'm curious why the pace of language is so slow
|
| This is a strength of the language.
|
| If you've ever done CI/CD for a large project in a fluid
| development ecosystem (e.g. nodeJS), you can understand why
| it might be refreshing to develop your operating system in
| a language where standards are measured by the decade.
| LAC-Tech wrote:
| yeah fair enough should've read the rest of the article
| before my snide comment
| kazinator wrote:
| C99 has one nice feature compared to C89 that is little talked
| about: you can initialize local aggregates using expressions that
| can't be calculated at load-time. void fun(int
| x) { struct foo = { x }; // not allowed in
| C89 int bar[3] = { 0, x }; // ditto }
|
| Chances are the kernel does this because, I think, it's also a
| GNU89 extension.
| marcan_42 wrote:
| Linux already uses a lot of C99 features that were GNU
| extensions before C99, like designated initializers. So in
| practice, Linux is already C99, and hasn't been buildable on
| strict C89 compilers in a long time.
| kwijibob wrote:
| Does C standard naming have a Y2K problem? :)
| teddyh wrote:
| Not until 2089.
| qiskit wrote:
| The C standards aren't published at a regular interval of 10
| years. They are published as needed or as agreed upon. So it
| is possible there may be a C standard published in 2089, but
| highly unlikely.
|
| It's similar to but not quite the same thing as the Y2K
| problem.
| jacobr1 wrote:
| So it is like a hashtable collision
| syncsynchalt wrote:
| Given C's ubiquity and (likely) longevity it'll have a Y2k89
| problem: standards released in 2089 and beyond will have to
| change their naming schema and/or avoid being standardized in
| particular years. ;)
| marcosdumay wrote:
| > or avoid being standardized in particular years.
|
| That's a very "legacy code" way of avoiding the problem, it's
| just a great option for a centenary frozen language.
|
| "Oh, no, look, C98 is from 2198. Do not confuse it with C97
| that was settled in 2097."
| thetic wrote:
| Though C89 does not support declaration of a scoped loop variable
| like C99 does: for(int i = 0; i < 10; i++) {
| // ... }
|
| C89 does support declaration of variables at the top of braced
| blocks whose scope is limited to that block: {
| int i; for(i = 0; i < 10; i++) { //
| ... } }
| cesarb wrote:
| That extra scope is an issue for what they're doing, however.
| They have a macro called "list_for_each_entry()" which
| syntactically behaves as if it were a "for(...)" (because it
| _is_ a "for(...)"). To make the extra scope work with that
| macro, either the user of the macro would have to end the loop
| with an unbalanced amount of closing braces (to match an extra
| opening brace within the macro), or it would require a second
| macro to be used at the end of the scope. Keep in mind that
| this macro is used in literally _thousands_ of files within the
| kernel.
| thetic wrote:
| Sure. This may not be a feasible approach for this case. I'm
| just pointing out alternative approaches to mitigate this
| kind of scoping problem in C89. Maybe wrap every use of
| list_for_each_entry in braces? {
| list_for_each_entry(...) }
| spc476 wrote:
| Then you get dinged by static analysis tools like SonarQube
| because you introduced a useless scope or increased the
| "cognitive complexity" to the code and have to explain it to a
| team leader who might not understand why you did that.
| jrockway wrote:
| I guess don't use a C99 linter on a C89 codebase?
| hn_go_brrrrr wrote:
| A team lead who does not understand that does not have the
| technical chops to be a team lead.
| lazide wrote:
| Doesn't mean they aren't still your team lead though!
| staticassertion wrote:
| Should we really be making decisions with people like
| that in mind?
| lazide wrote:
| If they are your team lead, would you have a choice?
| dale_glass wrote:
| One thing I've wondered for some time is why don't the Linux
| developers modify the compiler to suit their needs better.
|
| At that project scale, wouldn't it start making sense to start
| solving problems like "If it were possible to write a list-
| traversal macro that could declare its own iterator [...]" by
| adding the functionality you want to GCC?
| Jtsummers wrote:
| That would make the code (and the compiler) non-C89 compliant.
| At which point, why not move to a different standard rather
| than bork both the kernel and the compiler if it can meet the
| needs better?
| electroly wrote:
| The Linux kernel requires GNU extensions; it's already not
| ANSI C89.
| [deleted]
| Jtsummers wrote:
| Alright, making it _more_ non-compliant, then. Still
| creates problems for everyone and also necessitates a
| larger jump in the GCC version to support (unless you 're
| going to back port the new extensions to every version that
| the kernel currently works with). Currently you can compile
| the Linux kernel with GCC 5.1. If you changed the compiler
| to support this one feature (or any other) you'd either
| need to back port it several versions (potentially back to
| 5) or abandon all of them. Which is probably a no-go with a
| project that's overall as conservative as the Linux kernel
| project.
| pabs3 wrote:
| They do; IIRC have some GCC plugins to change various things.
| initplus wrote:
| Maybe that's helpful today, but by remaining (relatively)
| standards compliant the kernel codebase has a more long-term
| resiliency. The more tightly wedded to a single compiler
| implementation the greater the long term risk for the project.
|
| If some shenanigans were to happen upstream in GCC it would not
| be the end of Linux.
| colejohnson66 wrote:
| But they're _not_ standards compliant. They rely on many GCC
| extensions that make compiling on something else not
| possible.
| tinalumfoil wrote:
| Other compilers support GCC extensions.
|
| > The Linux kernel has always traditionally been compiled
| with GNU toolchains such as GCC and binutils. Ongoing work
| has allowed for Clang and LLVM utilities to be used as
| viable substitutes. Distributions such as Android,
| ChromeOS, and OpenMandriva use Clang built kernels. LLVM is
| a collection of toolchain components implemented in terms
| of C++ objects. Clang is a front-end to LLVM that supports
| C and the GNU C extensions required by the kernel, and is
| pronounced "klang," not "see-lang."
|
| https://www.kernel.org/doc/html/latest/kbuild/llvm.html
| jancsika wrote:
| Current situation-- do nothing but yell at the compiler devs.
| Benefit: sometimes they listen. Cost: you cannot always (or
| maybe even often) get the compiler to behave as you think it
| should because you don't control it.
|
| Deathtrap situation-- maintain an operating system _and_ a fork
| of a compiler. Benefit: you can get more control over the
| compiler. Cost: you _still_ cannot always get the compiler to
| behave as you think due to time constraints. Death cost: your
| first cost is multiplied by the fact that you 're now
| maintaining a goddamned compiler.
|
| I rankly speculate Linux is an extant project at its scale
| _because_ it has refused to fight on two fronts like this. (And
| yelling across a border isn 't the same as crossing it.)
|
| Edit: clarifications
| aliswe wrote:
| - and you will have a compiler which is out of date.
| kllrnohj wrote:
| GCC already has that functionality, it's called C++. Like that
| macro is just a crappy version of std::for_each.
|
| It doesn't seem useful to make a "C+" instead of switching to a
| language that just has the feature set the kernel needs. Like
| Rust, which is gaining some support within the Linux kernel
| already.
| brenns10 wrote:
| > Like that macro is just a crappy version of std::for_each.
|
| It really, truly is not comparable. list_for_each_entry does
| a well known operation that's just moving some pointers
| around. Safe to do while holding a spinlock, or in interrupt
| context, etc.
|
| std::for_each is a generic iterator implementation. What code
| runs in order to iterate? Will it transparently take locks,
| risking a deadlock? Will it need to schedule()? You don't
| need to ask those questions about the standard linked list
| macro, and that has a lot of value when every instruction the
| compiler generates might matter.
| midjji wrote:
| All for(auto x:xs){L} does is:
|
| { iter= xs.begin(); while(iter++ != xs.end()){L} } Not
| exactly hard to see what happens. In particular as iter is
| typically just a raw pointer.
|
| Regardless of if you use one or the other you must answer
| the question, Should I take locks, etc.
|
| With the macro you also have the concern that if any of the
| headers you include happens to change order of inclusions,
| the macro called may change without being noticeable. That
| is a harder problem.
| kllrnohj wrote:
| list_for_each_entry just does blind string substitution.
| std::for_each is the basic same thing except with type
| safety & variable scope safety instead. It's still just
| substitutions.
|
| All your other stuff about for_each is just... Wrong? It's
| all well defined what it does. The substitutions that
| for_each makes and exactly what it calls are all defined.
|
| You could give it a container that does something stupid,
| but you could also give list_for_each_entry the wrong field
| or accidentally use it when it's no longer valid (literally
| the bug being fixed)
| brenns10 wrote:
| Point taken on how for_each works for linked lists. I'm
| (obviously) not a C++ expert and it shows in that
| context.
|
| I suppose the thrust of my opinion is that a lot of code
| depends on the fact that it iterates over a linked list,
| which is safe in many contexts. The idea that callers
| could substitute container types which break its
| assumptions seems unwelcome to me. Nobody provides the
| wrong fields to list_for_each_entry because they get very
| nasty compiler errors if they try. But anybody can plug a
| new container type into a for each statement and get new
| behavior that silently breaks the assumptions of the
| original code.
|
| I suppose that could be protected by a better type system
| though. There's a lot to improve on C even for kernel
| dev, and "C is greatest" isn't a hill I'm willing to die
| on :)
| kllrnohj wrote:
| > But anybody can plug a new container type into a for
| each statement and get new behavior that silently breaks
| the assumptions of the original code.
|
| How would it be "silent"? You're changing the container
| at the point where you're making any assumptions about
| it. This would just fall under normal code review
| purview, just like code reviews are how Linux enforces
| that anyone even used this linked list macro in the first
| place. C certainly doesn't care if you use
| list_for_each_entry for your container iteration, after
| all.
|
| But it'd be a lot easier to replace these linked lists
| with a vector that has the same observable contract if
| C++ was used, just would be way faster to iterate on.
| mhh__ wrote:
| They've dug themselves into a hole when it comes to C++.
|
| If you started writing a major project in C now you'd
| rightfully get sacked, but open source projects like this are
| often dominated by the opinions of those who only work on
| that project. That doesn't mean they're automatically wrong
| but just that they're often massively detached from any
| feedback apart from disaster.
| stjohnswarts wrote:
| *for drivers primarily
| Gigachad wrote:
| That's basically the trial period. Nothing depends on
| drivers so they are pretty safe to try new things with and
| the hardware they are used for is limited so you can know
| you the hardware the driver is used for is supported by the
| Rust compiler. If we go a few years and its all good and
| the rust compiler supports everything linux targets, I can
| imagine more core parts becoming rust.
| dale_glass wrote:
| C++ was discussed as having too many undesirable
| characteristics.
|
| But that's exactly why I think a customized language for the
| kernel is an interesting idea. You could get pretty much
| exactly what you want for the kernel. Add features you need,
| remove any undesirable behavior or features.
|
| For most programs that would be too much complexity, but the
| kernel has very particular needs. And I think a similar in
| spirit approach has worked very well with Qt.
| kllrnohj wrote:
| Let's say you wanted to do that but also stay in the C
| family. It's going to be a _lot_ easier to take C++ and
| just ban all the stuff you don 't like (such as via lint
| rules) than to make a new language. In the former case you
| continue to benefit from all the ecosystem compiler &
| tooling improvements (ide support, static analysis,
| etc...). In the latter case you're on your own. Which is no
| small burden here.
|
| I'm not necessarily advocating that Linux _should_ switch
| to C++, just that there 's probably not a good reason to
| invest in a new C/C++ hybrid language at this point in
| time. Not when C++11 is honestly pretty good, but also
| Rust, Zig, or even D's betterC all already exist.
| tialaramex wrote:
| C++ has implicit allocation and Linus doesn't like that,
| for good reason. While it's amusing to imagine trying to
| "ban all the stuff you don't like" when that means core
| language features, it is not a realistic plan.
|
| In Rust, allocation lives in a library, alloc, and so the
| Rust for Linux project did all the work to offer alloc
| (it's full of useful stuff and it isn't like the Linux
| kernel can't allocate memory) but without implicit
| allocation.
|
| If you use Rust to write say a Linux command line
| program, you can write greeting += " and
| welcome traveller";
|
| ... and of course implicitly this is an allocation, 'cos
| it's not like this greeting variable just magically
| already has enough space to append a string. But in Rust
| for Linux, you can't do that, the implicitly allocating
| += operator is not provided on this type in their alloc
| library. If you want to say "Allocate more space for the
| greeting" you can do that of course, just as you can
| today in C but you must do so explicitly and so when you
| try to add 16GB of extra string space because you're an
| idiot, the API you had to explicitly call gives you an
| _error_ and that 's your problem.
|
| However the most critical reason Linux doesn't have C++
| is that C++ proponents didn't do the work. Now, that will
| probably be because "reform the entire language to suit
| Linus" wasn't a viable plan, but the fact is that Linus
| can't accept patches that nobody writes, so even if
| you're sure C++ would be viable without drastic changes,
| you didn't write the patchset that does it. Likewise if
| people don't send Linus patches to do C11 it probably
| won't happen.
| pjmlp wrote:
| Arduino, ARM mbed, Symbian, macOS, Windows, BeOS, IBM i
| and z/OS accepted the work of C++ proponents.
|
| There is nothing when can do to change political views.
|
| And lets not pretend that Rust in the kernel won't suffer
| from creative uses of macros, and there is still a big
| laundry list of issue to fix before it actually makes it.
|
| Or that for the time being, Rust compilers need to link
| to C++ code to actually work, at least until Cranelift is
| a match in code quality against LLVM or GCC.
|
| So in the end Linus gets C++ into the kernel, even if
| indirectly.
| oconnor663 wrote:
| > Rust compilers need to link to C++ code to actually
| work
|
| GCC itself has been written in C++ since 2010.
| pjmlp wrote:
| Indeed, which kind of diminishes Linus's assertions
| against C++ even more.
| kllrnohj wrote:
| > C++ has implicit allocation
|
| Ah, but see, it doesn't. Linus was wrong about many
| things in his rant. This being one of them.
|
| Now the standard library does indeed have things that do
| implicit allocations, such as std::string. But these,
| like with Rust, are distinct from the language &
| replaceable. Would it be effort to make a kernel-safe
| std:: replacement? Yes. Would it be a _lot_ of work? Not
| really. And it 's the kind of thing Linux has been doing
| for decades with C anyway. It's not like they use a
| standard libc implementation (much less a rich libc
| implementation like glibc), for example.
|
| And it's something game devs have been doing with C++ for
| decades without any issues, too.
|
| So for this one you don't even have to ban anything. Just
| don't pass a standard library implementation to the
| compiler. It doesn't come with one, after all. You have
| to add it. So you could just... Not do that.
|
| As for nobody did the work... That's true. But Linus also
| pretty much nuked the entire concept of using C++,
| regardless of the what or how. Time seems to have changed
| his mindset on some of those things, hence his reaction
| to Rust, which largely makes all this moot. But we
| shouldn't confuse short term politics with technical
| issues, either.
| Galanwe wrote:
| > Linus was wrong about many things in his rant. This
| being one of them.
|
| Its not what is being discussed here. C++ has an
| implementation defined memory model. That just cannot
| work with Linux out of the box. You need an OS and
| compiler designed from the ground up to handle this.
| kccqzy wrote:
| C++ has a well defined memory model:
| https://en.cppreference.com/w/cpp/language/memory_model
| And it's not that different from the C memory model. How
| did Linux work with C?
| Galanwe wrote:
| > C++ has a well defined memory model
|
| Only since C++11, and it's not a full blown memory model.
| gpderetta wrote:
| C++ has with a close, approximation, the same memory
| model as C, both concurrent and otherwise. In fact C11
| just adopted the C++11 concurrent memory model wholesale.
| jcelerier wrote:
| it's defined enough for C to have adopted it since C11
| (https://people.mpi-
| sws.org/~viktor/papers/cpp2015-invited.pd...)
| midjji wrote:
| Its notable that the main improvement over gnu98 C to C11
| essentially templates, if a worse version of it with
| better compile time performance.
|
| If they switched to c++17, and banned everything from
| throw and RTTI to the standard library, they would still
| get for each, auto, and templates and inheritance which
| does not have to be emulated in C.
| Koshkin wrote:
| The desirable characteristics of C++ overweigh, by far, its
| "undesirable" ones (whatever those may be, and which by the
| way you can for the most part safely ignore, if you wish).
| chaxor wrote:
| I don't know anything about C++ really, but I have been
| interested in learning it several times. The biggest
| deterrent for me has been the huge number of languages
| that you have to know to learn c++ it seems. Every new
| version comes with so many different features and
| seemingly (by looking for example repos on git) different
| paradigms that it essentially feels like many different
| languages.
|
| This is probably one of the biggest reasons I may turn to
| rust over C++ - simply less features creep due to less
| time being around.
|
| Can you comment on how wrong I am about my feelings this
| way?
| midjji wrote:
| Not wrong, but its a feature, rather than a problem.
|
| C++(17+) template based metaprogramming far more powerful
| and generic than what you can do in eg rust. Converting
| eg a rust or go or python, or julia, etc library into c++
| is pretty straightforward, just use appropriate
| overloads, and a few template tricks, and you can mostly
| copy the code directly. But copying between these, or
| from c++ is much harder.
|
| The solution is to learn one approach suitable for the
| problem you have right now, then widen over time. The
| many languages of c++ as you describe is less of a
| problem, the problem is that some approaches are deeply
| and fundamentally flawed, but still in common use.
| kllrnohj wrote:
| Learning/teaching C++ is definitely a sore spot about it.
| Many still teach C++98 which is... Bad. It'd be like
| still teaching Java 1.3. yeah the syntax is the same
| basic shape, but pretty much everything else is
| different.
|
| C++11 and newer are all largely one "category" in terms
| of recommendations. The CppCoreGuidelines is a good place
| to cover all that, but it's not a from-scratch
| introduction by any means
| Koshkin wrote:
| Yes, I understand, I would probably feel the same if I
| wanted to learn C++ all at once and for its own sake.
| Luckily, I have never had to do such a thing. Indeed, I
| may never learn C++ in its entirety, and I may even miss
| some "important" pieces, but I am OK with that. I am very
| comfortable using C++, and I still continue picking up
| pieces of wisdom here and there - as I go. My advice,
| learn by example, start with a small project and go from
| there.
| jcelerier wrote:
| > C++ was discussed as having too many undesirable
| characteristics.
|
| C++ allowed the Serenity OS people to produce an entire OS
| with a GUI stack able to play Diablo and their own web
| browser in something like two years. It's depressing to
| think where we could be today in terms of OS if the Linux
| and GNU people weren't as insistent on their hate of C++
| edgyquant wrote:
| You could build Serenity in C in a similar about of time
| akling wrote:
| Perhaps someone could, but I certainly couldn't. :)
| jcelerier wrote:
| What are the arguments towards this ?
| von_lohengramm wrote:
| Note that Serenity OS also has its own stdlib "AK" that
| excludes a lot of criticized C++ "features". If the Linux
| devs were to carefully splice out the Good from the Bad
| and Ugly of C++, then maybe it'd suit their tastes just
| fine. I think the bigger factor here is that Linux became
| decidedly anti-C++ back when C++ was actually pretty
| crumby.
| jcelerier wrote:
| The Linux kernel has its own libc-ish too.
| tayo42 wrote:
| Why is c++ necessary? Shouldn't any programing language
| be able to do the same things. Maybe with different
| amounts of code
| Koshkin wrote:
| Sure, as my prof once noted, "Why, with the assembler you
| just need to type more!"
| microtherion wrote:
| You must have had one of those progressive professors who
| did not consider assemblers "a waste of a valuable
| scientific computing instrument [...] to do clerical
| work": http://worrydream.com/dbx/
| reasonabl_human wrote:
| I love historical anecdotes like this. Goes to show that
| we should always be questioning our status quo and
| thinking bigger.
| mhh__ wrote:
| Because it's better.
|
| Simple question, simple answer. Using C is purely an act
| of risk-aversion, contrarianism, and fashion.
| josefx wrote:
| Please write a JavaScript program that will not deploy to
| your production system if you accidentally pass a string
| to any function that expects an integer.
| cprecioso wrote:
| So... typescript?
| xroche wrote:
| > C++ was discussed as having too many undesirable
| characteristics.
|
| I tend to agree, as a C++ developer. There are many core
| issues in the language that haven't been resolved and that
| are unacceptable for kernel code.
|
| My personal pet peeve: C++ is unable to reallocate a new[]
| region. This makes basically all structures (vector,
| hashmap, trees...) unusable for large data handling.
| pjmlp wrote:
| Google (C++17 subset in Zircon), Apple (IO Kit drivers
| with Embedded C++) and Microsoft (since Vista), ARM
| (mbed) IBM (C++ as PL/S replacement) think otherwise.
| kllrnohj wrote:
| Of those structures only vector benefits from realloc
| anyway. And if you really want it, it's not terribly hard
| to write. You can static assert that the template type is
| trivially movable which would make it safe to realloc.
|
| But this is assuming you have a malloc implementation
| that does something other than implement realloc as just
| malloc+memcpy+free. Which not many do, not unless the
| allocation is so large as to be in its own dedicated mmap
| or similar.
|
| That aside, sure would be great if you elaborated on
| these unsuitable, unresolved for kernel language issues?
| Exceptions and rtti are the only two I'm aware of and
| both have had off switches for decades.
| xroche wrote:
| > And if you really want it, it's not terribly hard to
| write
|
| Sure, but basically it means rewriting all structures
| that rely on a bucket of stuff.
|
| By the way maps often use a large bucket, and rehash in-
| place can be preferable.
|
| > Which not many do, not unless the allocation is so
| large as to be in its own dedicated mmap
|
| Do you know a modern operating system that does not have
| a mremap equivalent ?
|
| On Linux you pretty much use it as soon as you reach
| large blocks.
| kllrnohj wrote:
| > By the way maps often use a large bucket, and rehash
| in-place can be preferable.
|
| std::unordered_map (what I'm guessing you meant by a
| hashmap) uses a linked list for the nodes. There's no
| movement in the first place to worry about being
| realloc'd.
|
| > Do you know a modern operating system that does not
| have a mremap equivalent ?
|
| You have to be very large before most mallocs will put
| you on a dedicated mmap that can even be mremap'd at all.
|
| If you're working with stonking huge data inline in a
| std:: vector... Yeah just make a container for that
| usage, not really an issue. There's tons of examples out
| there, typically to add SSO but doing realloc would be
| the same basic thing.
| brenns10 wrote:
| They do! Consider this patch which adds the GCC implementation
| necessary to make "static keys" work:
|
| https://gcc.gnu.org/legacy-ml/gcc-patches/2009-07/msg01556.h...
|
| Static keys are a kernel API that allows code modification at
| runtime to achieve zero cost feature flags, like tracing:
|
| https://www.kernel.org/doc/html/latest/staging/static-keys.h...
| stjohnswarts wrote:
| because currently gcc maintenance is almost "free" (in
| comparison to maintain gcc which is as big a project as linux)
| and they just have to work with the powers that be over at gcc.
| That's much better than forking it and having yet one more HUGE
| project to maintain independently. The advantages of sticking
| to the current policy FAR outweigh the disadvantages.
| [deleted]
| na85 wrote:
| I'm not a compiler expert here but when clang was new and one
| of the BSD distros was making a lot of hay about moving to it,
| I remember one of the arguments in favor being that the GCC
| codebase is a mess and not easily extended.
| sfblah wrote:
| Couldn't they solve that macro issue using C89 by just having the
| macro create a block around the loop, and then declare the
| variable at the top of the block?
| Miiko wrote:
| No, because while macro can add the opening bracket for the
| block, it cannot add corresponding closing bracket.
| sfblah wrote:
| Makes sense. Thanks.
| pjmlp wrote:
| The reference to moving into C99 is kind of interesting, given
| the security work paid by Google to remove all uses of VLAs, I
| thought they were already using it anyway.
| dahfizz wrote:
| VLAs exist as a gnu extension even for -std=c89, unless
| -pedantic is also specified.
| pjmlp wrote:
| Ah, forgot about it. Thanks.
| userbinator wrote:
| I'm one of those people who think OS kernels should stay as
| portable and simple as possible (i.e. C89 or some other easily-
| bootstrappable language, to avoid Ken Thompson attacks), so this
| isn't great news to see "the ladder being pulled up another
| rung". Then again, Linux has already become immensely complex.
| tomcam wrote:
| > so this isn't great news to see "the ladder being pulled up
| another rung"
|
| Man I so vibe with that. But the new Cs do have a ton of new
| features and, more to the point, I trust Linus to make this
| kind of decision more than just about anyone.
| thestoicattack wrote:
| The article does mention at least one advantage, so it's not as
| if the ladder is being pulled up for no reason.
| pcwalton wrote:
| The advantages of using a newer language greatly outweigh the
| disadvantages of theoretical "reflections on trusting trust"
| attacks. By mandating C89, you're condemning thousands of
| kernel developers to use a language that's over 30 years old
| because of a theoretical attack that has never happened and
| seems practically implausible. Does anyone really think there's
| a backdoor in both GCC _and_ Clang (remember, Linux can be
| compiled on either)?
| gmadsen wrote:
| I am ignorant on nearly all compiler related issues, but
| there was an article on here not too long ago that was
| arguing that nearly all os development required old C because
| choices of the committee would break use cases required under
| the guise of undefined behavior.
|
| is the benefits of "modern C" worth compile times 2-3x times
| longer?
| aw1621107 wrote:
| > but there was an article on here not too long ago that
| was arguing that nearly all os development required old C
| because choices of the committee would break use cases
| required under the guise of undefined behavior.
|
| I would guess that you're referring to either "How ISO C
| became unusable for operating systems development" ([0]) or
| "How One Word Broke C" ([1]).
|
| [0]: https://arxiv.org/abs/2201.07845 , most recent HN
| discussion at https://news.ycombinator.com/item?id=30022022
|
| [1]: https://web.archive.org/web/20210307213745/https://new
| s.quel... , HN discussion at
| https://news.ycombinator.com/item?id=22589657
| pcwalton wrote:
| Times like these I wish I were allowed to say exactly how
| much money big companies save by using the newest versions
| of GCC and Clang. The economic value of modern compiler
| optimizations is staggering.
| tomcam wrote:
| I hereby give you permission to say exactly how much
| money big companies save by using the newest versions of
| GCC and Clang.
| pcwalton wrote:
| Thanks, but it's my employer's permission I'm concerned
| with. :)
| bch wrote:
| > wish I were allowed to say exactly how much money big
| companies save[...]
|
| Are you able to give hints that would guide us in thought
| exercises?
| bluGill wrote:
| Facebook has hinted that they employ smart C++ people
| because some core optimizations can save on the order of
| several hundred thousand dollars per year (possibly in
| the millions). Most of that is off because they don't
| have to buy as much power to cool the server rooms, some
| of it is buying less servers as what they have can do
| more. Facebook doesn't actually say what they save, but
| they have said they measure it, and we know how much it
| costs to employ an engineer full time working only on
| optimization, and we know who some of those engineers
| are.
|
| You have to be very large to notice it, but the likes of
| Facebook, amazon, and Google have massive warehouses
| almost entirely filled with computers. It doesn't take
| much to see how their power bill can add up.
| plorkyeran wrote:
| Years ago Alexandrescu said that Facebook estimated a .1%
| speedup to HHVM would save them $100k/year. It's
| presumably only gone up since then, but he's stopped
| giving an actual number in talks.
| pertymcpert wrote:
| One avenue is to think about it in terms of datacenter
| compute. If modern compiler optimizations improve
| performance by 10%, aggregate the perf improvements
| across the entire world's compute and you get a huge
| saving.
| ______-_-______ wrote:
| The Google Search optimization team measures changes in
| terms of millions of CPUs. So if upgrading your compiler
| saves even 0.1%, well, that's a lot of servers you don't
| have to buy next year.
| steveklabnik wrote:
| The parent works at Facebook/Meta leading their Rust
| team, focusing on compiler and ecosystem improvements. So
| that in and of itself is a sort of hint into what that
| info could be like, even if it's not actual details of
| the order of magnitude or anything.
| bonzini wrote:
| Why would a different standard cause longer compile times?
| db48x wrote:
| If the spec requires that the compiler look for a certain
| error and produce a diagnostic, then the compiler has to
| spend time doing that. The more complicated the spec is,
| the more work the compiler has to do to find the errors.
|
| For example, Rust is frequently said to have long compile
| times. There are a number of interesting reasons why that
| is often true, but if we ignore the pathological cases
| then what we find is that borrow checking takes a
| significant fraction of that compile time. This is a big
| trade-off between features and complexity that the Rust
| language made very deliberately: the advantages of the
| borrowing rules are what makes Rust such a great
| language. The cpu time spent checking that those rules
| have been followed are a small price to pay, but not a
| negligible one.
| midjji wrote:
| Is there a single such which you cannot disable by
| compiler flags?
| mananaysiempre wrote:
| Most of the really annoying UB is technically in C89
| though, it's just that compilers haven't really treated it
| with such contempt for the first two decades or so. I can't
| even recall any new kinds of (non-library) UB in C99 or C11
| (though there have to be some).
|
| So "old C" in such a case would need to mean "an old C
| _implementation_ " (or possibly a new one, but simple or
| configured to behave like an old one), something like GCC
| 2.8 maybe, and nobody's using that on desktop. So the
| language standard version should be mostly immaterial, and
| it's not like the C89-to-C17 difference is anything like
| the yawning C++98-to-C++20 chasm. (This is a carefully
| phrased statement: C99 had complex numbers, which are
| annoying, and variable-length arrays, which are a
| significant change, but C11 demoted both to optional
| features.)
| dataangel wrote:
| I think you're mixing up two different things. Newer
| standard versus newer compiler. All versions of the C
| standard have lots of undefined behavior. Newer compilers
| contain optimizers that are just better at leveraging it.
| neysofu wrote:
| > is the benefits of "modern C" worth compile times 2-3x
| times longer?
|
| I mean, you get more security by default and security is
| pretty darn important for kernels...
| pm215 wrote:
| Linux has always relied heavily on GCC extensions, though -- it
| makes no attempt to actually be C89-compliant portable code.
| What it actually has is a minimum supported gcc version, which
| in turn governs whether particular features can be used. In
| this case the minimum gcc version has for other reasons finally
| got big enough that C99 and C11 support is definitely present
| -- the ladder was already this high.
|
| (You can also build with clang, but only because clang
| deliberately aims to support most gcc extensions.)
| Dylan16807 wrote:
| Declaring a variable in the middle of a function takes
| basically no effort to support. Especially compared to all the
| ridiculous things the kernel gets up to.
|
| The ladder's being moved a millimeter.
| syncsynchalt wrote:
| IMO the C standards are conservative enough that even a
| standard level "only" a decade old is still fine.
|
| With that said, we still have options even when moving to a
| newer standard. Many new language features can be machine-
| translated to C89 if needed (similar to how we have protoize /
| unprotoize, though not always as seamless). If we need to keep
| a bridge to C89 the kernel authors could hold to a subset of
| new language features that are amenable to machine translation.
| jcranmer wrote:
| Linux is not written in C89. It is written in gnu C89, which is
| a mixture of C89, C99, and a panoply of often poorly-defined
| gcc-specific features. The number of compilers that can
| successfully compile Linux is one; not even clang is able to
| fully do so yet, I believe.
|
| Actually, as a compiler writer, I'd go a little bit further and
| point out that Linux itself isn't even written to the gnu C89
| very well; it's often written to a "C is portable assembly"
| view of the language, which results in nasty grams and
| invective being hurled at compiler writers if they compile the
| C specification correctly and not according to the "proper"
| assembly the code author thought they were getting.
|
| One of the benefits of more modern language revisions is that
| they actually tighten the wording on a lot of the more
| ambiguous parts of the specification--C11 in particular adds a
| much more comprehensive memory model that's very _shrug_ in the
| older revisions of C.
| nikanj wrote:
| My understanding is that Linux has often faced issues where
| "compiling the specification correctly" means "Ha-ha gotcha,
| we can actually throw half your code out of the window
| because you misread the deep aliasing rules on page 8432 of
| the spec"
|
| Their hesitancy with newer standards is understandable, when
| viewed against that backdrop
| jcranmer wrote:
| One of these days, I will get around to writing my post as
| to why that take on undefined behavior is completely wrong.
|
| More to the point, though, the only changes to undefined
| behavior in the C specification in newer versions (compared
| to C89) are either clarifying things that _were already
| undefined behavior_ (e.g., INT_MIN % -1) or actually
| _making some undefined behavior well-defined_ (e.g.,
| allowing type punning via unions).
| comex wrote:
| C23 will make calling realloc with a size of 0 into
| undefined behavior. [1]
|
| [1] http://www.open-
| std.org/jtc1/sc22/wg14/www/docs/n2464.pdf
| bonzini wrote:
| That falls under things that were already undefined or
| impossible to use properly before.
|
| Older standards said "it is implementation-defined
| whether the old object is deallocated". This didn't
| really work well:
|
| - if realloc(..., 0) returns NULL if it freed the object,
| then you have confusion with error cases. strtol already
| has this kind of interface and it's unusable
|
| - if realloc(NULL, 0) returns NULL and does nothing it
| does something different than malloc(0). Some chose to
| make it do something different, some chose consistency
| with malloc.
|
| - if you choose consistency with malloc then
| realloc(NULL, 0) likely will end end up inconsistent with
| realloc(ptr, 0) where ptr is not NULL. On BSDs the two
| are consistent but also different from any other
| platform, so portable code could not rely on realloc(...,
| 0) doing something known: either you had possible double-
| free bugs on some platforms, or you had a memory leak.
| plorkyeran wrote:
| I'm very unhappy that they chose to make it UB rather
| than mandating a behavior, but in portable C89 calling
| `realloc(ptr, 0)` is always a bug in your code, and it
| was an insane thing to declare implementation-defined
| originally.
| electroly wrote:
| Clang can do it. Google ships clang-built Linux kernels in
| Android and ChromeOS.
| seabrookmx wrote:
| Also the Kernel used in the new Valve Steam deck.
| lupire wrote:
| Can clang compile vanilla kernel without Google's forks?
| electroly wrote:
| Apparently OpenMandriva also uses a clang-built kernel,
| presumably not a Google fork. I have no personal
| knowledge about OpenMandriva though.
| elevader wrote:
| AFAIK it can. They actually spent quite a considerable
| amount of effort implementing all of the gcc extensions
| to do so. There is even documentation for that:
| https://www.kernel.org/doc/html/latest/kbuild/llvm.html
| karlding wrote:
| Do you consider the ClangBuiltLinux project [0] a Google
| fork?
|
| [0] https://clangbuiltlinux.github.io/
| gsnedders wrote:
| Clang 9.0+ & Linux 5.3+ work for x86_64; I believe it's
| been possible to compile arm64 for longer.
| mwint wrote:
| What's a Ken Thompson attack?
| [deleted]
| nl wrote:
| _Ken Thompson 's "cc hack" - Presented in the journal,
| Communication of the ACM, Vol. 27, No. 8, August 1984, in a
| paper entitled "Reflections on Trusting Trust", Ken Thompson,
| co-author of UNIX, recounted a story of how he created a
| version of the C compiler that, when presented with the
| source code for the "login" program, would automatically
| compile in a backdoor to allow him entry to the system._
|
| https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html
|
| The paper's a pretty entertaining read:
|
| > First we compile the modified source with the normal C
| compiler to produce a bugged binary. We install this binary
| as the official C. We can now remove the bugs from the source
| of the compiler and the new binary will reinsert the bugs
| whenever it is compiled. Of course, the login command will
| remain bugged with no trace in source anywhere.
| ksec wrote:
| Reflections on Trusting Trust - Ken Thompson
|
| https://wiki.c2.com/?TheKenThompsonHack
| Jtsummers wrote:
| https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref.
| ..
|
| Ken Thompson, _Reflections on Trusting Trust_.
|
| EDIT: I don't think I've seen 5 nearly simultaneous replies
| sharing the same link before.
| ksec wrote:
| >EDIT: I don't think I've seen 5 nearly simultaneous
| replies sharing the same link before.
|
| LOL I was searching for a non-PDF link and delayed the
| reply. It would have been 6 simultaneous answer.
| bombcar wrote:
| https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref.
| ..
|
| How do you prove your stack is secure?
| gmfawcett wrote:
| Solution: the compiler for the microlanguage at the bottom
| of the stack is none other than the kindly and
| incorruptible American film star, Tom Hanks.
| mindcrime wrote:
| I believe the person you are replying to is using that as an
| allusion to the issues discussed in the famous "Reflections
| on Trusting Trust" paper[1] by Ken Thompson.
|
| [1]: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984
| _Ref...
| [deleted]
| febstar wrote:
| Most likely referring to "Reflections on Trusting Trust" [0];
| i.e. when the compiler is itself the attack vector.
|
| [0]: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984
| _Ref...
| esarc wrote:
| https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref.
| ..
| wolrah wrote:
| > I'm one of those people who think OS kernels should stay as
| portable and simple as possible (i.e. C89 or some other easily-
| bootstrappable language, to avoid Ken Thompson attacks)
|
| Why does the complexity of the OS, or even its ability to be
| compiled by multiple compilers, matter for "trusting trust"
| attacks?
|
| As I've always seen it, the problems and solutions all exist at
| the compiler level. The whole premise relies on starting with a
| binary compiler that you are expected to trust. The source is
| also assumed to be safe and un-tampered for purposes of this
| discussion because that's an entirely different issue.
|
| The solution, of course, is to build the compiler itself with
| different compilers. If you build the same compiler with two or
| more different compilers, then use that to compile itself, you
| should be able to with the right options (see the work that has
| been done on reproducible builds) get binaries that are equal
| or close enough to easily compare any differences.
|
| At that point if they are the same then you know either it's
| good or both of the upstream compilers were also compromised.
|
| If for whatever reason that's not practical at the top level
| compiler the same concepts apply going further back in history
| until you get to some early compiler a bored grad student wrote
| in the '80s in pure ASM.
|
| As a result from a practical sense I don't really see "trusting
| trust" attacks to be that big of a concern. It's always
| possible to work your way back down the tree of software until
| you get to a point where you can actually trust a compiler and
| then build your way back forward from there.
|
| If a compiler starts depending on its own tricks and gets to a
| point where it can only be successfully compiled by itself,
| then there are reasons to be suspicious. Even then you'd just
| have to have the last version to be able to be built by other
| packages as one more stop along the way to trust.
| pinephoneguy wrote:
| Yes but Linux won't even build with gcc 4 much less weird
| compilers like tcc or msvcc. That ship has unfortunately
| already sailed.
| tiahura wrote:
| Sounds like GNU/Linux is probably warranted.
| yjftsjthsd-h wrote:
| Kind of, but you can also use clang. On the other other
| hand, that's clang with GNU extensions, so /shrug
| Zababa wrote:
| > to avoid Ken Thompson attacks
|
| I was always under the impression that the "message" of
| Reflections on trusting trust was that you have to trust
| someone at some point.
| neysofu wrote:
| Ken Thompson's hack is purely theoretical, especially in the
| modern computing landscape where the diversification of
| software supply chains would make such hack much 1. less
| effective and 2. easier to detect. If we're talking about
| kernel security, there are other lower hanging exploits, many
| of which are enabled by outdated language design decisions and
| unsafe memory models.
|
| As compiler technologies evolve, it becomes not only a
| necessary evil but rather a better course of action to trust
| compilers rather than humans tiptoeing around security risks
| masquerading as language idiosyncrasies.
| TheDesolate0 wrote:
| yjftsjthsd-h wrote:
| > Ken Thompson's hack is purely theoretical
|
| Erm. I was always under the impression that he had actually
| done it, and that his presentation was a historical anecdote.
| Is that not the case?
| neysofu wrote:
| Yes, he did, but that just shows that it's possible to
| inject self-replicating code in a compiler. It doesn't
| actually tell us how resistant such code is to compiler
| source changes, audits, binary analysis, etc. which are all
| required for a successful exploit.
| boogies wrote:
| But the University of Minnesota experiments prove at
| least some malicious code is capable of passing at least
| standard Linux code review, right?
| lonjil wrote:
| No, no malicious code passed. _All_ malicious code was
| rejected before making it into the kernel.
| josefx wrote:
| I think they only got past a single reviewer and counted
| that as success, none of their harm code made it into the
| kernel proper. However the University also committed
| thousands of automated "fixes" for linter warnings that
| weren't part of the study and the kernel maintainers
| ended up reverting all of them due to the Universities
| overblown claims.
| IshKebab wrote:
| That's a completely separate issue. The trusting trust
| bug requires you to convince people to use your compiler
| binaries. The thing you're talking about just requires a
| patch to pass code review.
|
| It's more work - you have to figure out a sneaky bug and
| write a legit looking patch - but anyone can do it. You
| don't have to be in a position of power already (e.g.
| being the Debian GCC packager) so overall it is _much_
| easier.
| gmfawcett wrote:
| There's a world of difference between that and Thompson's
| attack. You don't exploit the kernel code: you exploit
| the compiler code, such that every program it compiles
| (a) is compromised, and (b) is incapable of detecting the
| exploit in programs compiled by the same compiler.
| lupire wrote:
| Parent didnt say anything about patch code review.
| steveklabnik wrote:
| On a technical level, the hack has been demonstrated to
| work, but actually hacking someone via the technique has
| not, in my understanding.
| marcan_42 wrote:
| It's obvious that the hack would work on a technical
| level, that's fundamental to the whole idea. The question
| is how complex the logistics of doing it in the real
| world would be; how you insert the tampered code, how you
| avoid anyone noticing, etc.
|
| For example, I run Gentoo Linux, and haven't reinstalled
| my OS since 2004 or so. That means that, modulo a few
| binary packages, I have a direct source lineage to the
| state of Linux in 2004. If you want to pull off that
| attack against my system (and you didn't already back in
| 2004), you'd have to tamper with source archives. That
| would both imply changes that are easy to analyze (more
| than binary patches), and it would involve changing the
| archive hashes in the Portage tree. That tree is in Git,
| which means that it would create an immutable public
| record of what happened (Git is the original blockchain,
| remember), modulo forced pushes which people would,
| again, notice all over the place.
|
| In practice, if you want to persistently backdoor a new
| system (supply chain attack), it's usually easier to do
| that in hardware or firmware than trying to do a RoTT
| attack on the distro and its compiler. In fact, it's
| users of binary distributions (or proprietary OSes) that
| should be more worried, as it is _much_ easier to do a
| binary-based RoTT attack that self-updates to handle new
| versions consistently when all your users run the exact
| same binaries. Source code users should be more worried
| about compromise upstream than local persistence. And
| those attacks are a review / auditing issue, unrelated
| to RoTT.
|
| In the end, if _you_ are worried about being personally
| targeted, it 's easy enough to make that impractical by
| re-bootstrapping your computing from an unpredictable
| source (e.g. walk into a random shop and buy a PC, walk
| into a net cafe and download your favorite distro and
| check the hashes there). And if you are worried about
| large-scale attacks, RoTT style ones aren't practical
| without someone somewhere noticing; you should be worried
| about traditional compromise instead.
| dreamcompiler wrote:
| As far as we know.
|
| Thompson's hack relies on the Halting Problem, and the
| space for deviousness within the Halting Problem is
| infinitely large.
| truffdog wrote:
| Does https://en.m.wikipedia.org/wiki/XcodeGhost count?
| It's not targeting compiler developers, so it can't worm
| forever, but it is a malicious compiler that weaponizes
| its output.
| rurban wrote:
| In this case practical attacks are very likely, due to
| insecurities added with C11 and the stoneage review model of
| Linux development.
|
| C11 added insecure Unicode identifiers, and you won't find
| them in reviewing them manually via email. You need a special
| linter to detect such Unicode attacks. Or a proper
| development environment.
|
| Compiler development usually evolves into more bugs, not
| less. Just now they caught up with the hundreds of bugs they
| added with gcc-9. Not talking about const, restrict, and
| strict aliasing, and the still not existing -Oboring for the
| kernel. Would you dare to use -O3 and -flto and -fstrict-
| aliasing in the kernel?
| marcan_42 wrote:
| Linux has a checkpatch.pl. If it doesn't already test for
| Unicode identifiers, it would be trivial to add that. This
| is a non-issue. You don't ban a new version of a language
| just because it allows some undesirable things; it's
| trivial to ban those things specifically, either with
| scripting or specific compiler option overrides.
|
| Heck, Linux already uses GCC _plug-ins_ to implement some
| fancier security stuff; it is silly to think they can 't
| handle forbidding Unicode identifiers.
| mlindner wrote:
| You're confusing portability with age. Just because something
| is old does not make it more portable. It arguably makes it
| less portable as new platforms will not support old language
| versions.
| baby wrote:
| I find it scary that most modern infrastructure eventually relies
| on C, and C from decades ago at that.
| lazide wrote:
| You shouldn't - it isn't perfect, but it is well understood. So
| far we don't have anything else that works better and all the
| edge cases and weirdness is understood by enough people that
| someone could build something like Linux on.
|
| It is unwise to build a bridge on something untested with
| unknown failure modes, and it is equally unwise to do a rewrite
| or create new core infrastructure in a language without knowing
| it as well as a civil engineer knows concrete.
| Koshkin wrote:
| I hear you. And a huge codebase at that. Unfortunately, the
| choice is limited; D or Ada would probably be better
| alternatives today.
| freedomben wrote:
| The amount of fear I see from people around C is so surprising
| to me. Not saying you fit into this, but so far my anecdata
| suggests that it's largely people who don't know (or know very
| little) C. Especially among CS grads whose main exposure to C
| was in the stack smashing exercise in a security class, all
| they know about C is the part that was intentionally made
| vulnerable so it would be easy to exploit. Unless you're being
| wildly negligent and reckless with your programming, C is
| really not that scary.
| viraptor wrote:
| And yet, the most significant part of the C code issues we
| find is memory corruption. Which is either significantly
| harder to cause or impossible-by-design in many alternatives.
| Unless you can realistically say "people working daily, for
| years, on huge C projects write those bugs, but they're
| reckless and I'm better than them" - yes, C should be scary
| to you these days.
| enneff wrote:
| What's scary is the huge number of bugs in the Linux kernel
| that wouldn't exist if it were written in a safer language.
|
| https://syzkaller.appspot.com/upstream
| jimbob45 wrote:
| https://stackoverflow.com/questions/50724726/why-didnt-
| gcc-o...
|
| We can't even agree on safe string functions for C, half a
| century later. You shouldn't have security bugs baked into
| the standard library and you shouldn't have to do a mountain
| of research to know which functions are safe and in which
| cases.
|
| However, for most things non-string, non-pointer, and non-
| array, I agree with you.
| baby wrote:
| I mean even an off by one is a security issue
| rurban wrote:
| We don't even have a string library in C. Strings are
| Unicode, not just zero-terminated buffers. You cannot find
| strings, nor compare them. In the kernel you have
| filesystems and login systems using unidentifiable names.
| Because the kernel has no identifier support.
|
| And for insecure standards, the committees rather want to
| eliminate the safe functions, than fixing the spec bugs or
| add u8 support.
| Koshkin wrote:
| > _We don 't even have a string library in C._
|
| What prevents us from having one?
| baby wrote:
| How would you mess with parsers by inserting null bytes
| in your strings then? (Remember seeing that in a talk
| from moxie on x509 certificate parsing.)
| freedomben wrote:
| It's easy to say "You shouldn't have security bugs baked
| into the standard library" but it's a lot harder to say,
| "we're breaking decades worth of working code by removing
| some functions that have been part of the standard lib and
| were widely used."
| mhh__ wrote:
| Skilled C programmers make these mistakes day in day out, we
| write too much code to trust a language that just let's it
| happen. Especially one that encourages the writing of code
| multiple times rather than reuse.
| baby wrote:
| My fear comes from the fact that I was a security consultant,
| and reviewed a lot of C applications containing nasty bugs.
| yjftsjthsd-h wrote:
| What percentage of non-C applications did you review? I
| suspect C has enough footguns to be an issue, but its
| popularity, especially in low-level software (kernels,
| codecs, firmware) ensures that it'll show up in security
| issues regardless of how bad the languages itself is.
| baby wrote:
| I reviewed mostly C, then reviewed mostly Golang. But I
| also reviewed codebases in Erlang, Perl, Java, C++, Rust,
| Python, Javascript, etc.
|
| Mind you I was mostly reviewing cryptographic-related
| applications, but most C applications contained bugs that
| had nothing to do with the logic (lots of memory
| corruption bugs) while most Golang applications contained
| logic bugs. Or at least I would find logic bugs because
| Golang was both rid of most memory corruption bugs, and
| also an extremely readable language (so easier for me to
| understand the code and find logic bugs). Although Golang
| still had nil dereference bugs (happens a lot when people
| used protobuf), because they don't have sum types. Today
| I think a great language would be a mix between a
| readable language like Go (with good defaults, toolings,
| stdlib) and a safe languages like Rust.
| j-krieger wrote:
| There is a significant margin between C applications and
| those of languages with a GC, at least in my experience.
| netr0ute wrote:
| How much GC stuff have you seen in comparison to C?
| baby wrote:
| There's a difference in GC languages though. strongly
| typed languages like Golang will always be more secure
| than dynamically typed languages like Python.
| kwertyoowiyop wrote:
| Definitely. Though Linux code has probably been tested more
| thoroughly, and run through more static analysis, than any
| other C code base. That does help me sleep a little better
| at night.
| freedomben wrote:
| Interesting, I had the same job in mid to late 00s,
| although I wasn't a consultant so my sample was the
| company's codebases (of which there were a lot because we
| built a lot of embedded systems on top of vxworks that did
| a lot of network communications, sometimes in very niche
| protocols), not necessarily the codebases of company's that
| are worried enough that they hire a consultant. That was
| right around the time when compilers and security tools
| were becoming available that could flag nearly every
| possible problem. At that point false positives became a
| big challenge.
|
| What years were you a consultant reviewing C applications?
| baby wrote:
| I'm guessing you were using tools like coverity? I
| actually never used such tools. I mostly did manual
| reviews and sometimes implemented fuzzers with AFL. But
| most of the code I looked at was crypto code. Did that at
| Matasano/NCC Group from 2015-2019
| freedomben wrote:
| it's been 15 years so I don't remember the names of the
| tools, but coverity rings a bell. There was one that we
| used to make fun of a lot because it was written in Java,
| but it was by far the best at finding stuff. It would
| even show you the AST to help point out problems. I'm
| suddenly feeling really nostalgic about GUIs written in
| Swing and SWT :-D
| Banana699 wrote:
| Apparently Mozilla, Microsoft, Google, IBM and all companies
| slowly moving their system infrastructure to Rust don't know
| enough C.
|
| >Unless you're being wildly negligent and reckless with your
| programming, C is really not that scary
|
| https://blog.regehr.org/archives/970
| warmwaffles wrote:
| > Apparently Mozilla, Microsoft, Google, IBM and all
| companies slowly moving their system infrastructure to Rust
| don't know enough C.
|
| They are moving that direction because of the safety
| promise that Rust offers and it is a "hot" language with a
| bunch of momentum behind it.
| nikanj wrote:
| This reminds me of
| https://www.usenix.org/system/files/1311_05-08_mickens.pdf
| P_I_Staker wrote:
| I don't know about that. I agree about it being a popular
| idea with people that don't understand C. Plenty do and still
| harbor those opinions. I've been mortified by what I've seen
| in C, and it seems really preventable.
|
| Lots of developers out there are widely negligent. C provides
| them with enough rope to hang themselves. To be honest, I'm a
| little surprised to find veteran C developers that AREN'T
| scared. I guess they just see every disaster as the fault of
| "negligence and recklessness". If you're not scared of your
| code (mistake), you should certainly be scared of other
| peoples.
| pornel wrote:
| I've been programming in C for 20 years, and I think it's
| scary. Not because I don't know it, but because I know
| exactly how many cases of seemingly normal code can hide UB.
| I know from experience that even the best programmers
| following best practices will make mistakes (or run into
| someone else's). C is an extraordinary amplifier of bug
| severity. I know how much diligence, effort, and tooling it
| takes to merely not screw things up in C.
|
| I've seen time after time people saying "nah, C is fine, you
| just avoid this and that, use these tools, etc." and this
| turning out to be insufficient. I've heard many times "maybe
| you're just a bad programmer and can't handle C, but I'm a
| good programmer and have no problems" and their code not
| surviving 5 minutes of fuzzing. I've seen people conclude
| that multi-threading beyond simplest constructs is just
| infeasible to get right, and think that's an inherent
| property of threading, and not fragility of C.
| P_I_Staker wrote:
| Yeah, I find it astonishing to find C programmers that see
| no problem, even though I think it would be reasonable to
| say that they see no reason to change. It's mind boggling.
|
| Have they not worked on large projects with other
| developers? Have they not seen the myriad of ways things
| can silently go wrong for seemingly no technical benefit?
| (although I know there's often less obvious reasons, for
| eg. UB, performance, platform specificity, etc.)
|
| All that, said I do think the following might be
| reasonable: > "nah, C is fine, you just avoid this and
| that, use these tools, etc."
|
| I guess you mean they write off the risks entirely? You
| should never be this "handwavy", and should always take the
| risk seriously, especially with a language like C. However,
| I think it's fair to say, that C is a good choice, many of
| the risks can be mitigated, and it's not THAT big a deal.
| In which case, the above doesn't seem that absurd.
|
| Following basic common sense and making an effort to
| identify and eliminate some of the sketchier situations,
| backed by some really good integration testing, can really
| help. I feel reasonably safe under those circumstances (I
| mean not really, but other languages can be "unsafe" too).
| A huge chunk of the really evil things I've seen have been
| the result of taking absurd risks, and/or disregarding the
| rules entirely. If you were paying attention and "trying"
| to write good C code, they would never happen; these aren't
| just individual developer things, but project wide. Eg. I
| had a compiler that didn't even warn for implicit
| functions... jerk move by TI, but that should be flagged
| and dealt with. Instead, the team just thought "great no
| compilation errors".
|
| I will say there's a lot of developers that are just
| uninterested in any of this and will deliver some really,
| really sketchy C code. In their mind they're smart
| programmers and their code will just be right, and they
| don't seem to understand any of these issues. Just plow
| ahead and patch around the bugs, then move on to the next
| gig.
| ylk wrote:
| This isn't the Linux kernel but I'd say it's fair to assume
| that the same likely applies to it:
|
| > Most of our memory bugs occur in new or recently modified
| code, with about 50% being less than a year old.
|
| > [...] we've found that old code is not where we most urgently
| need improvement.
|
| https://security.googleblog.com/2021/04/rust-in-android-plat...
| tombert wrote:
| That makes some intuitive sense, right? The fact that it got
| "old" in the first place indicates that it's not being
| touched a lot, and if it isn't being touched a lot that means
| that bugs haven't been found, meaning that the bugs that are
| in there are especially sneaky edge cases, or there simply
| aren't any large bugs to begin with.
| P_I_Staker wrote:
| This is kind of one of those issues that can really cut
| both ways. I don't think it's the best attitude to say,
| "it's been working for years, so it's fine"; there could be
| subtle bugs, and areas rarely exercised. Still it often
| holds true. We all remember times we've meddled with
| something and messed it up. It seems that some of the low
| level code has been really heavily used in many different
| ways, and seems to just work. Especially if it's not
| safety/security critical (and maybe even if it is), it
| could be a poor use of resources trying to fix something
| that isn't broken.
| treesknees wrote:
| I had this argument with some of my management. There was a
| push to upgrade all of our libraries that were deemed "old"
| with no other criteria than "it's outdated." I can
| appreciate shiny new toys, but if you're not hitting bugs
| and things are stable, I'd rather put my effort into adding
| features to our codebase and not chasing down library bugs.
| baby wrote:
| There's more risk in not updating dependencies due to not
| patching bugs
| tombert wrote:
| I haven't really touched non-GC'd languages in quite awhile,
| but I feel like modern C isn't _that_ unsafe, at least from the
| bits I 've played with it; it can even have a garbage collector
| if you want it [1](which I usually do).
|
| It's worth giving it another try if you haven't in awhile, if
| for no other reason to understand what's going on behind the
| scenes of your abstractions in Java/C#/JavaScript/etc.
|
| [1] https://en.wikipedia.org/wiki/Boehm_garbage_collector
| Koshkin wrote:
| > _isn 't that unsafe_
|
| It is as unsafe as you let it be, consciously or by mistake.
| int_19h wrote:
| This is still valid modern C: long* p =
| malloc(sizeof(int));
| jahlove wrote:
| What language is more unsafe than C? C++? ASM?
| CyberRabbi wrote:
| Wouldn't it be the normal course of events that infrastructure
| would be based on decades old established technology...?
| M_Tf_442 wrote:
| I actually find it reassuring. C is the most mature and proven
| language out there. With proper standards in place (best
| example probably NASA), pitfalls can be avoided. Another
| example that immediately comes to my mind is Redis, which is
| such a great and stable piece of software. Java is similar
| mature, but not the right tool for infrastructure.
| TheDesolate0 wrote:
| rrauenza wrote:
| I'm trying to understand how this linked list works --
| struct list_head { struct list_head *next, *prev;
| }; struct foo { int fooness;
| struct list_head list; }; struct foo
| *iterator; list_for_each_entry(iterator, &foo_list,
| list) { do_something_with(iterator); }
|
| If we are walking iterator->list->next ... how do we get the
| pointer of the next enclosing foo struct? Are they doing pointer
| arithmetic to get the beginning of the struct from the list field
| offset and casting it to foo?
|
| Ah -- That does seem like what they do: #define
| list_for_each_entry(pos, head, member) \
| for (pos = list_entry((head)->next, typeof(*pos), member); \
| &pos->member != (head); \ pos =
| list_entry(pos->member.next, typeof(*pos), member))
| #define list_entry(ptr, type, member) \
| container_of(ptr, type, member) #define
| container_of(ptr, type, member) ({ \ const
| typeof( ((type *)0)->member ) *__mptr = (ptr); \
| (type *)( (char *)__mptr - offsetof(type,member) );})
| kragen wrote:
| Yeah, standard hack for intrusive linked lists in C and similar
| languages.
| mlindner wrote:
| Anyone who insists on C89 is just talking nonsense. There are
| zero downsides to C99 and if you are using C89 you should not be.
| asveikau wrote:
| ~7 years ago you needed c89 to port to MS's compiler. But they
| brought some of that up to speed.
|
| I would not be surprised if there is some niche environment
| somewhere that needs c89. Not any mainstream desktop, server or
| phone though.
| yyyk wrote:
| C99 added support for VLAs which are a mistake. Fortunately,
| Linux stopped using VLAs a while ago.
___________________________________________________________________
(page generated 2022-02-25 23:02 UTC)