Post B5FpFKCYBnqjR4lspc by abucci@buc.ci
(DIR) More posts by abucci@buc.ci
(DIR) Post #B5A36VQpIboY4MNVr6 by ngaylinn@tech.lgbt
0 likes, 0 repeats
I thought this was a particularly good analysis of the problem of using LLMs for science. It explores the purpose of science, the perverse incentives that drive people to use LLMs, and the impact this has on skill building and training future scientists.My lab group has been struggling with this topic lately, without much consensus. This blog post captures a lot of our thinking, and very clearly made some good points that we appreciated. It mostly just describes the mess we're in without offering much useful advice, but just laying out the problems do nicely is helpful. That said, I do worry the author may be underestimating the impact these tools might have on experienced researchers.https://ergosphere.blog/posts/the-machines-are-fine/#academicchatter #llm
(DIR) Post #B5A36ViYEhzaxKffU0 by abucci@buc.ci
0 likes, 0 repeats
@ngaylinn@tech.lgbt What's the struggle exactly? "Don't use LLMs in science" is the only stance one can reasonably take under current circumstances unless one is OK with eating everyone's seed corn. Any use of it degrades what science is and has immoral and unethical side effects. The only rationales I've ever seen offered for using LLMs in science boil down to responses to self-destructive, warped practices in places like the US today. These practices are not permanent, though. They will change, and then what? Our seed corn is gone.I found the article you linked poorly argued and full of alarmingly shallow framings. Here are a few bits and pieces I picked up, for what it's worth (warning: it's too long!):The models are already powerful enough to produce publishable results under competent supervision. That's not the bottleneck. The bottleneck is the supervision.Besides being false and circular, this is a poor framing that attempts to carve out one piece of a complex system when it is irrational to try to do that. This author does not seem particularly incisive about what exactly science, and artifacts like publications, are for. He seems close to getting it at times, commenting on the loss of competence that results, but then somehow confesses he's doing the very thing that'd lead to that outcome.Ban-and-punish violates academic freedomNo it doesn't? I haven't looked but I bet at least one person argued that banning pipe smoking inside of academic buildings violated academic freedom too. Murder and pedophilia are also banned-and-punished (some horrible people try to argue the latter is a violation of academic freedom, even today). It almost sounds like he's using "freedom" in the right-wing/libertarian sense of having as few constraints on behavior as possible.is unenforceableYes it is? Nobody smokes a pipe inside of university buildings anymore. Current professors face all sorts of punishments and censures, new ones yearly it seems, for a variety of things ranging from not using Canvas properly to not reporting expenses properly to not filling out this or that form exactly the right way. They are forbidden from doing all sorts of things. All of this is enforceable, and is enforced, sometimes brutally. The above statement sounds childish and self-serving in light of reality. It also has a libertarian flavor.asks early-career scientists to compete with one hand tied behind their backs while tenured faculty quietly use Claude in their home offices...which tenured faculty won't do if such policies exist and are enforced? This is just not a serious argument."Compete" is a neoliberal economics term that really has no place in a communitarian endeavor like science. More right-wing/libertarian flavor.I use AI agents regularlyEesh, he's already melted his brain with the brain melting machine ( https://arxiv.org/abs/2604.04721 ). Too bad, but this partly explains the superficial reasoning: this article is a confession, among other things, and operates in an emotional register.When I see junior PhD students entering the field now, I see something different. I see students who reach for the agent before they reach for the textbook.So many articles like this scoldingly blame students for the failures of their seniors. He's confessing to a self-fulfilling prophecy: he uses software agents because he thinks he's somehow immune to their harms (he's not), and in the same breath decries that students use them too and this is very bad. Seniors should model the behavior they want to see represented in their disciplines, not scold the youngsters.He's telling on himself that he doesn't consider himself a student or lifelong learner. Yet if he is not doing fundamentally boring research, he's in the same position intellectually as a newcomer to the field and will suffer from LLM use in the same ways they do, just with a different set of preconditions.Using an LLM as a sounding board: fine.No it's not? Science is a group collaboration, and speaking with a colleague as a sounding board advances the field for both people. Spitting ideas into a chat window and responding to whatever it spits back separates the individual from their colleagues and contemporaries, robbing themselves, their colleagues, and their field of enriching experiences and foreclosing on the development of good communication habits.I'm not arguing that LLMs should be banned from research. That would be stupidNo it wouldn't? Why would it be stupid to remove the accelerant from the campfire before we know whether accelerating the fire would send it out of control and burn down the campground?Schwartz can use Claude to write a paper because Schwartz already knows the physics. His decades of experience are the immune system that catches Claude's hallucinations. A first-year student using the same tool, on the same problem, with the same supervisor giving the same feedback, produces the same output with none of the understanding.Hmm, I think this points at where his logic/framing breaks. You can't ever separate the output from the understanding. Those are fully interwoven. The paper is evidence of the understanding that was earned through the process of producing the paper. A paper-shaped blob of text that superficially resembles that evidence cannot be that evidence.The tell here is that he thinks there's a difference between senior and student, but there is not unless you assume the senior is doing fundamentally boring work.And here is where I have to be fair to Bob, because Bob isn't stupid. Bob is responding rationally to the incentives he's been given."Responding rationally to incentives" is the language of neoliberal economics, which is an important factor in the dysfunctional practices that have led us here. Given the insanity of the times, and the co-option of the word "rational", being irrational is the more reasonable way forward.Anyway, it doesn't make sense for me to keep beating up this person's thing. I hope you and your labmates dig deeper than this in your explorations!
(DIR) Post #B5FpFJWMie6bKF2C4u by ngaylinn@tech.lgbt
0 likes, 0 repeats
@abucci Honestly, I'm with you. I boycott LLMs. It's not even slightly difficult for me. They're harmful and unethical, but also they solve a problem I don't have. I'm a fluent and prolific writer / coder, who enjoys craft, expression, and hard work.I wish it were as simple for my peers. They are very tempted by LLMs, and not dissatisfied with the results. I think it stems from feelings of inadequacy. Publish or perish. English as a second language. Coding imposter syndrome. Job prospects. FOMO. Whatever. They feel a strong temptation, despite knowing all the down sides.I push back where I can, and I think that's having a positive impact. My PI even put a moratorium on gen AI for now. But I don't think telling people they're simply wrong is a productive way of changing their behavior. I do not agree with or endorse all the things in this article, but I do sympathize with the author, and I think he does a good job of explaining the sort of conflict my lab and many others are struggling with right now.
(DIR) Post #B5FpFKCYBnqjR4lspc by abucci@buc.ci
0 likes, 0 repeats
@ngaylinn@tech.lgbtI think he does a good job of explaining the sort of conflict my lab and many others are struggling with right now.I've observed an alarming tendency among people who've already made it in academia to dissuade, scare away, or haze others, when they actually have the power to change academia for the better. They are empowered to refuse the push for AI from their university administration. They have the power to introduce modules into their classes to explore any and all dimensions of LLMs to critically examine this technology. They can make phone calls or email their colleagues on their students' behalf to help them land positions or funding or what have you. In short, they have the power to change all the conditions that are leading junior scholars to question whether they need to be using LLMs and to feel fear about their future prospects---they can largely alleviate that fear and settle that question in the negative. I think a lot of senior scholars are falling short, or outright failing, at this, and many blame their juniors for it. I've seen this many times, both privately and expressed in public articles, and I find it offensive. Obviously there are plenty of good senior people in academia, but there are far too many like this too. I suspect the author of the article you shared of being of the not-great variety.
(DIR) Post #B5G3dOh2XVHffuPC1w by ngaylinn@tech.lgbt
0 likes, 0 repeats
@abucci Completely agreed. I find the amount of learned helplessness in tenured professors pretty alarming.In this case, the PI is choosing to let the discussion and ultimate policy emerge from discussion with the group, rather than simply putting his foot down.I like that he's trying to get more opinions and make the whole lab feel involved. However, this same tendency when taken too far can be an abdication of responsibility. It's a bit of a pattern with him: conflating freedom and autonomy of his students with never providing them strong guidance based on his much greater experience.We also have never attempted to codify what values the lab works by, which makes it hard to talk about whether use of LLMs aligns with those values. At least this discussion is raising some of those value questions.
(DIR) Post #B5G3dP0BOKb2dHMTrs by abucci@buc.ci
0 likes, 0 repeats
@ngaylinn@tech.lgbt Right, that's a good point. Having the conversation about values is...uh...valuable! It's interesting that you are having this conversation in your lab, and maybe that's something to be hopeful about. I don't recall having conversations like this at DEMO lab. My recollection from my time there is that we absorbed some values by osmosis and made up others as we went along, mostly individually.
(DIR) Post #B5HgwvT304RACA7hcu by ngaylinn@tech.lgbt
0 likes, 0 repeats
@abucci That's mostly how we do it, too. ;)
(DIR) Post #B5HgwvmXpa27AdFH16 by wrench@wandering.shop
0 likes, 0 repeats
@ngaylinn @abucci This is a very nice thread. Thank you both.> The tell here is that he thinks there's a difference between senior and student, but there is not unless you assume the senior is doing fundamentally boring work.In particular, I really liked this.I have recently decided to explicitly tell the other members of my lab my positions on LLMs and my expected positions for the Lab members.I personally feel that it is hard to do it without becoming a 24/7 advocate, though...
(DIR) Post #B5Hgwvyb6lfrm0stns by abucci@buc.ci
0 likes, 0 repeats
@wrench@wandering.shop @ngaylinn@tech.lgbtI personally feel that it is hard to do it without becoming a 24/7 advocate, though...Yes. The pro-LLM drumbeat is constant and loud. Lots of money behind it. My personal stance is that it's important enough to resist being fully overrun by this technology that I'll invest whatever time I can doing so. I have a PhD in computer science, after all, and what does the P even mean, if not that? I know that other people have different constraints, and this is definitely not meant to suggest other people should do what I've chosen to do.