Post B4s0EWSTZLRWrW0Bvc by mirabilos@toot.mirbsd.org
(DIR) More posts by mirabilos@toot.mirbsd.org
(DIR) Post #B4qzaK4lzbS9iYmpmq by aral@mastodon.ar.al
1 likes, 4 repeats
So Anthropic employees are using Claude Code to contribute AI-generated code to open source repositories and hiding the fact using their own internal “undercover mode”. Totally trustworthy people.(Any open source project that at the very least requires disclosure of AI-authored contributions should immediately ban Anthropic employees on principle.)#AI #Anthropic #ClaudeCode #subterfuge
(DIR) Post #B4r3uTUT6DI6WZfbSy by stux@mstdn.social
0 likes, 1 repeats
@aral All the while they can't even mange to keep their code indoorsWhat a joke company
(DIR) Post #B4rVUsrovN3OPrB86K by aredridel@kolektiva.social
0 likes, 0 repeats
@dalias Morally arguable but not actually true under the copyright regime that exists. At what point does learning from others constitute their authorship?
(DIR) Post #B4rVUt9XrTERIpTHjE by IngaLovinde@embracing.space
0 likes, 0 repeats
@aredridel @dalias > but not actually true under the copyright regime that existsUnder the copyright regime that exists in the US specifically, the generated code is at best not copyrightable at all (and therefore cannot be included into any projects with licenses relying on copyright).Of course maintainers of said projects might decide to yolo it, but also they might decide to not; and in this case, the intentional deception by antropic becomes even more significant fraud.
(DIR) Post #B4rW60m4UKGNeeC2jo by IngaLovinde@embracing.space
0 likes, 0 repeats
@itamarst @aredridel @glyph it's not even just 4x as much; every MR requires 4x (or more) as much effort as a human written one, because the modes of failure are completely different. For human written MRs a general heuristic of "if it looks good, it's good" is applicable to some extent, but LLMs are optimized to generate code that "looks good" and that makes reviewer's eyes glaze and that passes the review successfully, regardless of its actual quality.
(DIR) Post #B4s0EW9ghCPjvFDBdw by aredridel@kolektiva.social
0 likes, 0 repeats
@aral Honestly I don't actually hate this.It's a tool. The _user_ is responsible for what they're submitting. It's putting code generated by them in their name. I think this is actually good.
(DIR) Post #B4s0EWSTZLRWrW0Bvc by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel @aral it’s not a tool.cf. https://toot.cafe/@baldur/114410495463114687
(DIR) Post #B4s0SaGEEDm0i2yiu0 by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel @dalias it is true.And LLMs cannot learn. They are merely a lossy compression/decompression thing. They regurgitate a somewhat averaged completion of the prompt from the other works they ingested.
(DIR) Post #B4s0Y4maAzfwoqjMXI by aredridel@kolektiva.social
0 likes, 0 repeats
@dalias Right. You're appealing to a definition of "FOSS" that isn't entirely clear what it is. And the people who do usually have (some) claim to that authority, the common uses of it, are not the ones you're using.I'm sympathetic to that but I can't tell what it is in an appeal to an unstated norm for a community that I can't quite identify.
(DIR) Post #B4s0Y4xvUokXO22QDY by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel @dalias that’s just deflecting, asking dalias to define something that is not even important to the point he’s making
(DIR) Post #B4s1OKfANWtX86BBnk by aredridel@kolektiva.social
0 likes, 0 repeats
@dalias Got examples to show to support that position? Remember copyright is a _legal_ regime and the legal regime seems quite oriented toward that NOT being the case.
(DIR) Post #B4s1OKszY7xBoyeELo by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel @dalias the Text and Data Mining exception to copyright law only applies ⓐ to models for analytics (discovery of patterns, trends and correlations; §44b UrhG), and ⓑ to works whose right holders didn’t opt out (ibid. p.3); there’s absolutely no basis on which “genAI” even could be considered permissible, as both the reproduction (§16) and the right to make changes, editions and other derivatives (§23) are protected by law, by default, and always require a licence.So, thrice denied, by existing law.
(DIR) Post #B4s1X56BcCYTSqzQAK by glyph@mastodon.social
0 likes, 0 repeats
@aredridel @aral I really can’t agree with this, because it’s a question of accurate labeling not of “responsibility” or “authorship”. co-authored-by is perhaps the wrong method for labeling such things, but consider raw milk. ultimately, it is indeed the producer’s responsibility to ensure their product is free of contamination. but disclosure of its method of production is explicitly the kind of requirement that allows consumers of said product to make safe choices
(DIR) Post #B4s1X5bjitnB2hkcLI by xgranade@wandering.shop
0 likes, 0 repeats
@glyph @aredridel @aral This. It's critical to understanding the risk presented by a PR to know that it's not authored by a human, but extruded and then at most reviewed by a human.From a quality perspective, and to your point earlier, code review is insufficient to find flaws in extruded code. While the legal risks are likely small, as per previous discussions, they're not strictly zero, nor are they well-tested in courts.
(DIR) Post #B4s1X5y4Nrem9yCS9Y by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@xgranade @glyph @aredridel @aral I’d even go so far and say that you cannot rightly publish the result of slop under a FOSS licence because you lack what in the Linux community is known as the DCO.
(DIR) Post #B4s1XAFaSg2tRSQV0q by xgranade@wandering.shop
0 likes, 2 repeats
@glyph @aredridel @aral I approach the above from what I consider a deeply unethical viewpoint, namely one of asking about the practicalities of slop PRs instead of the ethicality, in part because that's the only axis along which there's any nontrivial question whatsoever.Along an ethical axis, the onus is very strongly on OSS projects to reject slop PRs, at which point the Claude Code prompt is even more odious — it demonstrates an intent to lie in order to bypass codes of ethics.
(DIR) Post #B4s1e5pDJURpkBOVbU by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@xgranade @glyph @aredridel @aral and to bypass consent.
(DIR) Post #B4s1iL72PtyYJckxOq by aredridel@kolektiva.social
0 likes, 0 repeats
@mirabilos @aral Wow I straight up think this concept of 'tool' is not good. The stick, the bit of whatever to open a bottle? That absolutely is a tool. To have it have a purpose in advance is such a restriction on the idea of a tool.An enabling technology. A way to extend ourselves. That's a tool. Whether bag or bicycle.
(DIR) Post #B4s1iLRbBSQFLONNRo by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel @aral no, it’s not a tool. It’s a… I’m not English, so I don’t have all the words, best I can say a quick hack barely good enough for the use, but not designed and engineered to help the use. (I don’t want to use the word “hack” here because it’s worse than that.)
(DIR) Post #B4s1xlasF5scn04HdA by aredridel@kolektiva.social
0 likes, 0 repeats
@mirabilos Not, strictly, true, though I get what you're going at.There's a few phenomena going on that shape these tools beyond that.- Emergent complexity - "Memory" records- Embedded context- Incorporating social inputsI still think they are strictly tools, but ones that can self-adapt with alarming power if you configure them right. They have a medianizing effect on a lot of their output (that's actually one of the reasons they're good at code. We generally want code to be "normal". It's one of the many reasons it's pretty bad for more artistic creative work, morally and technically.)But that's not the same as only repeating the median. The temperature, the randomness injected in makes them actually jump to stuff that is at times nonsensical but also at times clever. It's just randomness, but then with a heap of context and congruence applied that is rather interesting.
(DIR) Post #B4s1xlqpHmdlaTX1Um by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel that’s all not learning because learning is not something an algorithm can do
(DIR) Post #B4s22WQ65NXlB0ILI0 by aredridel@kolektiva.social
0 likes, 0 repeats
@mirabilos Quick hacks are tools! That's the beauty of being human. We turn all kinds of stuff into tools.
(DIR) Post #B4s22WmQkLPMIGkB6G by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel no, they are not tools, you use them until you can engineer a proper tool, at best.
(DIR) Post #B4sj5EqIBtvxYh5aj2 by IngaLovinde@embracing.space
0 likes, 0 repeats
@dalias @mirabilos @aredridel which still means that at best the generated code cannot be copyrighted, and at worst it violated copyright and license terms of the original authors (whose works were ingested to train the model). In both these cases, the resulting code cannot be incorporated into any FOSS project with any license.Typically when people submit code to FOSS projects, they also (implicitly or explicitly) claim that they hold the copyright on the submitted code, and agree that this code will be licensed under the license the project uses (which they only have power to do if the first claim is actually true).When LLMs are used to generate code, the first claim is false, and it _is_ a contamination.
(DIR) Post #B4sspD1MC8XgokGAmO by aredridel@kolektiva.social
0 likes, 0 repeats
@IngaLovinde @dalias That's the thing. If it matters _tell the people submitting PRs_. The tool is just a tool (a capricious annoying frustrating tool) but it's the _people_ doing this that need to be accountable.
(DIR) Post #B4sspDE7QgkbSKEMfg by IngaLovinde@embracing.space
0 likes, 0 repeats
@aredridel @dalias but in this specific case, the people submitting PRs are the one that created the tool.We're talking specifically about a tool developed by Anthropic, which has a separate mode for Anthropic employees, which purposefully is "operating undercover" and creating MRs that mislead OSS maintainers about provenance of these MRs.
(DIR) Post #B4stIFmP15hdlu7A0m by aredridel@kolektiva.social
0 likes, 0 repeats
@IngaLovinde Huh I don't find this at all. It looks like a featureless soup — that ‘eyes glaze over', I guess, is a fail to me.
(DIR) Post #B4stIFy6Jb3oMBaVFI by aredridel@kolektiva.social
0 likes, 0 repeats
@IngaLovinde Actually backing up, I think that's where I'm already a little sketched out by it. “looks good, probably is good" is how a lot of the supply chain attacks have slipped in.
(DIR) Post #B4stIGA9amhYxZE824 by IngaLovinde@embracing.space
0 likes, 0 repeats
@aredridel and that's one of the reasons why we have a web of trust of some kind, and changes by first-time contributors deserve extra scrutiny, and nobody would accept a huge new feature or a huge refactoring from an unknown first-time contributor.And still most of the time one can expect that contributors, even the first-time ones, are acting in good faith and can be reviewed in good faith as collaborative contributors, not as adversaries who purposefully try to slip a vulnerability past code review, purposefully writing it in such a way that it looks plausibly like benign code.With LLM-generated code, code reviews should treat it as written by an adversary _every_ time. And reviewing code written by an adversary consumes much much more effort than reviewing code written by collaborator in good faith... and why would one even spend any effort on reviewing code written by adversary, when discarding that code and closing the MR is an option?
(DIR) Post #B4stcUhrEGlZrEdkfI by aredridel@kolektiva.social
0 likes, 0 repeats
@IngaLovinde Are there any examples of misleading out there?Or is it _just not mentioning it_?
(DIR) Post #B4stcUsUajH0ODcFF2 by IngaLovinde@embracing.space
0 likes, 0 repeats
@aredridel when a developer submits code, the default assumption is that they wrote it (and not, say, plagiarized it from somewhere without actually understanding it).And on the original screenshot, it's clear that not only do they not mention the actual provenance themselves, but that they go extra mile to ensure that it doesn't leak in any other way."You are operating UNDERCOVER", "do not blow your cover", "NEVER include [...] any [...] attribution" communicates intent very clearly and is a very clear admission of guilt, regardless of whether these magic instructions to LLM actually work or not.
(DIR) Post #B4su6sgwcJTYvyhco4 by IngaLovinde@embracing.space
0 likes, 0 repeats
@mxey it would be acceptable for inclusion into public domain projects. But most/all FOSS _licenses_ depend on the code being copyrighted; no GPL or MIT etc licenses can apply to public domain uncopyrightable code.
(DIR) Post #B4swQBlQ8BMrUo89pY by aredridel@kolektiva.social
0 likes, 0 repeats
@IngaLovinde There is no magic. Seriously. Models are just ... kinda bad, actually.
(DIR) Post #B4swQC11CBqQHBQc8u by IngaLovinde@embracing.space
0 likes, 0 repeats
@aredridel "magic" as in those who write these instructions follow the magical thinking that there is something to give instructions to, instead of just autocomplete engine to which instructions and data are passed in a single combined (not separated) blob of text.
(DIR) Post #B4tEQer9lQrYtqBeYS by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@dalias @IngaLovinde @aredridel if something cannot be copyrighted and no others’ rights apply, then it is in the public domain. For LLM output, which has been proven to vastly resemble existing code under copyright, that’s not the case.
(DIR) Post #B4tHzW0qw5lBTB2FEW by aredridel@kolektiva.social
0 likes, 0 repeats
@dalias Have you seen how people perform on similar tests?
(DIR) Post #B4tHzWAmLBhRxxgAhk by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel @dalias people are still humans, not machines.Are you a TESCREAList?
(DIR) Post #B4tJTGxLa2GBPk9H6G by aredridel@kolektiva.social
0 likes, 0 repeats
@mirabilos Not even remotely TESCREAList. However, I think it's a fair question to ask: why are we drawing lines how we do? Especially when comparing work product.
(DIR) Post #B4tJTH7Gz8CRuWnCZU by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel on the most basic level because copyright mandates human creativity, the expression of human personality
(DIR) Post #B4tNP0fSb9EpYS0NI8 by aredridel@kolektiva.social
0 likes, 0 repeats
@mirabilos The thing is that actual use of these systems tends to involve LOTS of human creativity and attention. Lots of video and bits get spilled on hierarchical autonomous agent models and the hype, but real use? Much more hands on. The "I don't write code by hand anymore" people aren't just a minority but an extreme minority.
(DIR) Post #B4tNP0uhgTQoJj8Y3E by mirabilos@toot.mirbsd.org
0 likes, 0 repeats
@aredridel the prompt is but one of the many inputs that go into the regurgigated thing, but a minority compared to the "training data" *shrug*
(DIR) Post #B4ua0b8TMpfTyrE1OC by IngaLovinde@embracing.space
0 likes, 0 repeats
@illogical_me @aredridel I'm not saying that you should not really review long time contributors code. I'm saying that regular _thorough_ review (and in my experience, what most people in corporate setting typically do is _way_ less than that) is nowhere near enough the kind of review that's needed to catch sophisticated adversaries.xz is a small project at least. But when you're working on a typical corporate project that has hundreds of lines changed per person per day, and not all people are good at reviewing even regular code: those who are good at reviews simply won't have enough time in the day to review all the code in "adversarial" mode with extra scrutiny.
(DIR) Post #B4ubEHMBniXoe7jEBM by illogical_me@mastodon.cloud
0 likes, 0 repeats
@IngaLovinde @aredridel When people describe corporate environments, I’m often shocked. You’ve worked in nicer places than I have. If someone accepts a basic code review comment like ‘you copy pasted this instead of extending the function,’ rather than shopping for another reviewer, I call that a win. That may be why AI doesn’t upset me much: with moderate feedback, I get better results from it than I often did from humans.
(DIR) Post #B4ubfTX8dX5uEJomK8 by illogical_me@mastodon.cloud
0 likes, 0 repeats
@IngaLovinde @aredridel anyway my point was not that doing adversarial reviews is easy, but that doing them only to new contributors gives a false sense of security. The xz story proves that sophisticated attackers are aware of this and use it against the OSS community. Again, this is not an LLM problem. It’s just a difficult problem.
(DIR) Post #B4ucPBfzMU5SARkrZo by IngaLovinde@embracing.space
0 likes, 0 repeats
@illogical_me @aredridel another way to optimize adversarial reviews is to send the author home every time you see even the slightest imperfection or questionable or just not very clear part in their code, which is a path Linux kernel takes afaik. Linux kernel is also infamous for being very hostile towards first-time contributors, despite approaching everybody the same regardless of whether it's their first contribution or not. If you introduce Linus-style code reviews in your project, then the problem of LLMs being adversarial will be easily solved by (almost?) all LLM-generated code being rejected outright on merit. Also nobody will contribute to that project.
(DIR) Post #B4uchZ5v22bQ0he9cu by IngaLovinde@embracing.space
0 likes, 0 repeats
@illogical_me @aredridel > If someone accepts a basic code review comment like ‘you copy pasted this instead of extending the function,’ rather than shopping for another reviewer, I call that a win.Sounds like your place doesn't really have any code reviews, only complete cargo cult of code review process.Thankfully none of the places I worked at were like that. Although most places I worked at before 2012 didn't have a code review process in any form, not even a cargo cult one (probably because before 2012 it wasn't that well known).