https://lwn.net/SubscriberLink/854645/334317047842b6c3/ LWN.net Logo LWN .net News from the source LWN * Content + Weekly Edition + Archives + Search + Kernel + Security + Distributions + Events calendar + Unread comments + ------------------------------------------------------------- + LWN FAQ + Write for us User: [ ] Password: [ ] [Log in] | [Subscribe] | [Register] Subscribe / Log in / New account An update on the UMN affair [LWN subscriber-only content] Welcome to LWN.net The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net! By Jonathan Corbet April 29, 2021 On April 20, the world became aware of a research program conducted out of the University of Minnesota (UMN) that involved submitting intentionally buggy patches for inclusion into the Linux kernel. Since then, a paper resulting from this work has been withdrawn, various letters have gone back and forth, and numerous patches from UMN have been audited. It's clearly time for an update on the situation. The writing of a paper on this research [PDF] was not the immediate cause of the recent events; instead, it was the posting of a buggy patch originating from an experimental static-analysis tool run by another developer at UMN. That led developers in the kernel community to suspect that the effort to submit intentionally malicious patches was still ongoing. Since then, it has become apparent that this is not the case, but by the time the full story became clear, the discussion was already running at full speed. The old saying still holds true: one should not attribute to malice that which can be adequately explained by incompetence. On April 22, a brief statement was issued by the Linux Foundation technical advisory board (TAB) stating that, among other things, the recent patches appeared to have been submitted in good faith. Meanwhile, the Linux Foundation and the TAB sent a letter to the UMN researchers outlining how the situation should be addressed; that letter has not been publicly posted, but ZDNet apparently got a copy from somewhere. Among other things, the letter asked for a complete disclosure of the buggy patches sent as part of the UMN project and the withdrawal of the paper resulting from this work. In response, the UMN researchers posted an open letter apologizing to the community, followed a few days later by a summary of the work they did [PDF] as part of the "hypocrite commits" project. Five patches were submitted overall from two sock-puppet accounts, but one of those was an ordinary bug fix that was sent from the wrong account by mistake. Of the remaining four, one of them was an attempt to insert a bug that was, itself, buggy, so the patch was actually valid; the other three (1, 2, 3) contained real bugs. None of those three were accepted by maintainers, though the reasons for rejection were not always the bugs in question. The paper itself has been withdrawn and will not be presented in May as was planned. One can, hopefully, assume that UMN will not be pursuing similar lines of research anytime soon. Patch re-review One immediate result of the attention drawn to UMN's activities was a loss of trust in its developers, combined with a desire in some quarters to take some sort of punitive action. Thus, one of the first things that happened when this whole affair exploded was the posting by Greg Kroah-Hartman of a 190-part patch series reverting as many patches from UMN as he could find. Actually, it wasn't all of them; he mentioned a list of 68 others requiring manual review because they do not revert easily. As it happens, these "easy reverts" also needed manual review; once the initial anger passed there was little desire to revert patches that were not actually buggy. That review process has been ongoing over the course of the last week and has involved the efforts of a number of developers. Most of the suspect patches have turned out to be acceptable, if not great, and have been removed from the revert list; if your editor's count is correct, 42 patches are still set to be pulled out of the kernel. For those 42 patches, the reasoning behind the revert varies from one to the next. In some cases, the patches apply to old and presumably unused drivers and nobody can be bothered to properly review them. In others, the intended change was done poorly and will be reimplemented in a better way. And some of the patches contained serious errors; these definitely needed to be reverted (and should not have been accepted in the first place). A look at the full set of UMN patches reinforces some early impressions, though. First is that almost all of them do address some sort of real (if obscure and hard to hit) problem; there was a justification for writing a patch. While many of these fixes showed a low level of understanding of what the code was doing and thus contained errors, it seems unlikely that any of them were malicious in their intent. That said, there are multiple definitions of "malice". To some of the developers involved, posting unverified patches from an experimental static-analysis tool without disclosing their nature is a malicious act. It is another form of experiment involving non-consenting humans. At a minimum, it is a violation of the trust that is required for the kernel's development community to work effectively. The 42 bad patches out of 190 is a 22% bad-patch rate. Chances are, a detailed review of 190 patches from almost any kernel developer would turn up a few that, in retrospect, were not a good idea. Hopefully that rate would not approach 22%, though. But it must be said that all of those patches were accepted by subsystem maintainers throughout the kernel, which is not a great result. Perhaps that is a more interesting outcome than the one that the original "hypocrite commit" researchers were looking for. They failed in their effort to deliberately insert bugs, but were able to inadvertently add dozens of them. Meanwhile, there is still the list of patches that did not revert cleanly. That list has not been posted publicly, but Kroah-Hartman did start with a subset of seven of them. He also noted that the TAB will be publishing a full report of the audit of all these patches once it is complete. Thus far, none of these patches have actually been reverted in the mainline; that seems likely to happen toward the end of the 5.13 merge window. Lessons learned One of the key lessons from this series of events would clearly be: do not use a free-software development community as a sort of free validation service for your experimental tool. Kernel developers are happy to see new tools created and -- if the tools give good results -- use them. They will also help with the testing of those tools, but they are less pleased to be recipients of tool-inspired patches that lack proper review and an explanation of what is going on. Another lesson is something we already knew: kernel maintainers (and maintainers of many other free-software projects) are overworked and do not have the time to properly review every patch that passes through their hands. They are, as a result, forced to rely on the trustworthiness of the developers who submit patches to them. The kernel development process is, arguably, barely sustainable when that trust is well placed; it will not hold together if incoming patches cannot, in general, be trusted. The corollary -- also something we already knew -- is that code going into the kernel is often not as well reviewed as we like to think. It is comforting to believe that every line of code merged has been carefully vetted by top-quality kernel developers. Some code does indeed receive that kind of review, but not all of it. Consider, for example, the 5.12 development cycle (a relatively small one), which added over 500,000 lines of code to the kernel over a period of ten weeks. The resources required to carefully review 500,000 lines of code would be immense, so many of those lines, unfortunately, received little more than a cursory looking-over before being merged. One final lesson that one might be tempted to take is that the kernel is running a terrible risk of malicious patches inserted by actors with rather more skill and resources than the UMN researchers have shown. That could be, but the simple truth of the matter is that regular kernel developers continue to insert bugs at such a rate that there should be little need for malicious actors to add more. The 5.11 kernel, released in February, has accumulated 2,281 fixes in stable updates through 5.11.17. If one makes the (overly simplistic) assumption that each fix corrects one original 5.11 patch, then 16% of the patches that went into 5.11 have turned out (so far) to be buggy. That is not much better than the rate for the UMN patches. So perhaps that's the real lesson to take from this whole experience: the speed of the kernel process is one of its best attributes, and we all depend on it to get features as quickly as possible. But that pace may be incompatible with serious patch review and low numbers of bugs overall. For a while, we might see things slow down a little bit as maintainers feel the need to more closely scrutinize changes, especially those coming from new developers. But if we cannot institutionalize a more careful process, we will continue to see a lot of bugs and it will not really matter whether they were inserted intentionally or not. [Send a free link] ----------------------------------------- (Log in to post comments) An update on the UMN affair Posted Apr 29, 2021 15:06 UTC (Thu) by willy (subscriber, #9762) [ Link] I'm not entirely convinced that Hanlon's Razor applies to someone who already bragged about punching you in the nose. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 15:55 UTC (Thu) by yann-gael.gueheneuc (guest, # 151961) [Link] "UMN will not be pursuing similar lines of research anytime soon." Hopefully, they will continue their research but in a different, collaborative manner. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 20:37 UTC (Thu) by mpg (subscriber, #70797) [Link ] I had the same reaction to this sentence: I think this is a very interesting line of research and I hope some researchers will pursue this, with better ethics and stronger methodology, and as you say, in a collaborative manner: experimenting _with_ the community rather than _on_ people without their informed consent. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 16:58 UTC (Thu) by blackwood (subscriber, #44174) [Link] Yeah in this case it seems to very much indicate a penchat for playing things fast&loose, which seems to be supported by the higher bug rate in patches from this group overall compared to maybe the kernel at large. I think what applies here is that generally it's best to trust people first and assume benevolent intent. But then ensure that good intent is socially enforced by playing tit-for-tat. Which is also what happened, even if patches from UMN will land again in the future, they will definitely be subjected to a lot more scrutiny and need to clear a higher bar of usefulnees than fairly "trivial" changes to mostly unused code in very old drivers. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 22:58 UTC (Thu) by Paf (subscriber, #91811) [Link ] In their (sigh) defense, the UMN didn't brag about it - it was this specific group of researchers. The hunt through every patch ever submitted by the U of M (ever!) seems ... frankly, a little silly though not harmful, and the idea of a blanket revert straight up a self inflicted wound with no useful purpose at *all*. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 15:37 UTC (Thu) by jkingweb (subscriber, #113039) [Link] Sober analysis, Mr. Corbet. Thank you for thee update. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 17:26 UTC (Thu) by intgr (subscriber, #39733) [ Link] > For those 42 patches, the reasoning behind the revert varies from one to the next. > In some cases, the patches apply to old and presumably unused drivers and nobody > can be bothered to properly review them. > The 42 bad patches out of 190 is a 22% bad-patch rate. I don't think it's fair to call all of these patches "bad" if in some cases they will be reverted only for the reason that nobody can be bothered to review them. In other cases, the patches touched some piece of code that was indeed found to be buggy, but did not entirely address the brokenness. I found many examples of this, e.g. https:// lore.kernel.org/lkml/b43fc2b0-b3cf-15ab-7d3c-25c1... : > This looks like a good commit but should be done now in a different way > - using pm_runtime_resume_and_get(). Therefore I am fine with revert > and I can submit later better fix. A far more interesting metric would be, how many of these patches were actually found to introduce a bug that wasn't present before? Looking at a random sample of the patch reviews suggests it would be quite low. Even the one patch that triggered this whole reaction (https:// lore.kernel.org/linux-nfs/20210407001658.2208535-...) was actually harmless, but added some redundant code. I'd like to nominate this whole drama for the "overreaction of the year" title. :) Also: shouldn't LWN issue some sort of retraction or update to the previous article? It implied that these pathces were intentionally buggy or malicious a few times, which this article admits was unfounded. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 18:29 UTC (Thu) by deater (subscriber, #11746) [ Link] > I'd like to nominate this whole drama for the "overreaction of the year" title. :) this particular issue is complex as it affects three areas. The Linux-kernel reaction is pretty well summarized in this article. There's the IEEESSP (Oakland) issue where a paper like this never should have made it into a "top" conference, especially as it turns out the authors were doing questionable things like modifying the abstract after the paper was already submitted, and apparently their conclusions in the paper were extremely misleading when actually compared to the actual methodology that was released later. How did this pass peer review? Sadly modern academic publications with double-blind reviews are well-known to be easily vulnerable to peer-review attacks (see the ISCA'19 incident). The final issue is the academic side of things at UMN. Typically the university would not care at all about this (though since the people involved do not have tenure they're on thinner ice than normal). Possibly UMN is even happy as their CS department is getting a lot of publicity out of this. Universities will put up with all kinds of academic dishonesty, abuse of students, etc, as long as the researchers bring in grant money (again, see ISCA'19, also Ullman winning the Turing award). However the wildcard here is the UMN researchers listed $800k of National Science Foundation grants on the withdrawn paper. NSF has been on a *huge* ethics kick recently. I'm assuming this is why UMN is responding so strongly, there's a chance this funding will be removed, or worse, their whole school will get audited. Sadly money talks these days :( [Reply to this comment] Retraction Posted Apr 29, 2021 19:45 UTC (Thu) by corbet (editor, #1) [Link] So, I just went back and reread the original article, but I find nothing there that seems to be in need of retraction. It was a report on what was happening at the time, and the article itself took no position with regard to the status of the other patches. If there is something specific there that you think needs to be retracted, please point it out to us. [Reply to this comment] Retraction Posted Apr 29, 2021 20:21 UTC (Thu) by Homer512 (subscriber, #85295) [Link] The retraction is likely in response to the unethical practice of human experimentation on the kernel devs without prior consent. Fruits from the poisonous tree, so to speak. Whether this is an overreaction is up for debate. [Reply to this comment] Retraction Posted Apr 29, 2021 20:29 UTC (Thu) by corbet (editor, #1) [Link] I think I must not have been clear, sorry. The comment that I was replying to was asking LWN to retract our previous article on this topic; I was saying that I don't understand why that would be necessary. [Reply to this comment] Retraction Posted Apr 29, 2021 20:46 UTC (Thu) by bronson (subscriber, #4806) [ Link] Calling for it to be retracted also seems eligible for an overreaction award. [Reply to this comment] Retraction Posted Apr 29, 2021 21:43 UTC (Thu) by mpg (subscriber, #70797) [Link ] So, I also went back and reread the original article. While I don't think a retraction would be in order, I'm afraid I still find this article less balanced than is usually the case here on LWN. To be fair, I don't think the article stated anything wrong, and my concern is more about the weight given to various "sides" of the story or various hypotheticals. Take this sentence from the first paragraph for example: "The patch [...] was duly questioned [...], but it is not an honest mistake; according to Kroah-Hartman, there has been an attack of sorts underway [...]". The way the semi-colon is placed makes it look (at least to my eyes, not a native speaker) like "it is not an honest mistake" is a statement of fact rather than Greg's opinion. Perhaps a more accurate wording would have been: "but, according to KH, it is not an honest mistake and there has been [...]". Also, I think the order in which different points of views / hypotheses are presented matters. For example it's only until the end of the article (after a few paragraphs about the reversal process) that we learn the researchers claim that none of the hypocrite commits made it to the kernel. The original paper is linked early in the article and described as detailing "the process of introducing use-after-free bugs into the kernel [...]". After re-reading the paper's "Ensuring the safety of the experiment" paragraph, I think a more accurate description would have been "details the process of _nearly_introducing UAF bugs into the kernel [...]". Of course the link to the paper is right there at the beginning of the article, so I can click and go read the paper right away. But more likely, I'll finish reading the LWN article first, and by the time I get to the end, I'll probably subjectively give more weight to the idea / hypothesis that intentional bugs made it to the kernel, perhaps in large numbers, and things might still be going on, than to the idea that no intentional bug was introduced in the kernel, and that there only were 3 hypocrite commits. I'm afraid the previous paragraphs read like I'm picking at details in the article. I'd like to clarify that my intention is not to attack this article, but rather to try and explain why I found it less balanced than the average LWN article in a more specific (and hopefully constructive) way than "that's just how it felt to me". Also, I want to emphasize that in general I really appreciate the way LWN reports in a balanced and distanced (as in "looking at the big picture" and "cool-headed") way on all sort of topics, including those that generate heated discussion. It's only because the average quality of LWN articles is so high that I can feel perhaps this one was slightly below the usual standards on that front. [Reply to this comment] Retraction Posted Apr 29, 2021 21:48 UTC (Thu) by intgr (subscriber, #39733) [ Link] I had to re-read the article as well, and while it's dissatisfying that it picks a side, you're right that it's mostly factually correctly reporting on Linux developers' reaction. For the record, I did not mean that the article as a whole needs to be "retracted", but I guess "acknowledgement" that the assumptions reported in the article turned out to be false. I'm sorry, "retraction" was too strong a word There is one statement that appears to be made in the voice of LWN: > In fact, a thorough review of patches emanating from UMN over the last year or two is probably in order for other projects, particularly high-profile ones. As published in the "hypocrite commits" paper, the known malicious patches had actually come from Gmail addresses, and we know now that suspected patches from UMN were not intentionally buggy. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 19:27 UTC (Thu) by agrawal-d (subscriber, # 141386) [Link] I noticed that Jonathan Corbet himself is a member of the Linux Foundation Technical Advisory Board. It is amusing to see he had to link to the ZDNet article to refer to the notice sent to UMN - perhaps he could not have shared it himself as a member of the board but was aware of the contents nonetheless. [Reply to this comment] An update on the UMN affair Posted Apr 29, 2021 21:46 UTC (Thu) by chfisher (subscriber, #106449) [Link] Although it is true that one should never attribute to malice that which can be adequately explained by stupidity or incompetence, as Jerry Pournelle once pointed out, the toleration of certain levels of stupidity or incompetence constitutes malice by itself. It appears that UMN has reached those levels. [Reply to this comment] Copyright (c) 2021, Eklektix, Inc. Comments and public postings are copyrighted by their creators. Linux is a registered trademark of Linus Torvalds