https://www.lesswrong.com/posts/tqmQTezvXGFmfSe7f/how-much-are-llms-actually-boosting-real-world-programmer This website requires javascript to properly function. Consider activating javascript to get access to all site functionality. LESSWRONG LW Login ProgrammingSoftware ToolsAI Frontpage 116 [ Question ] How Much Are LLMs Actually Boosting Real-World Programmer Productivity? by Thane Ruthenis 4th Mar 2025 4 min read A 1936 116 LLM-based coding-assistance tools have been out for ~2 years now. Many developers have been reporting that this is dramatically increasing their productivity, up to 5x'ing/10x'ing it. It seems clear that this multiplier isn't field-wide, at least. There's no corresponding increase in output, after all. This would make sense. If you're doing anything nontrivial (i. e., anything other than adding minor boilerplate features to your codebase), LLM tools are fiddly. Out-of-the-box solutions don't Just Work for that purpose. You need to significantly adjust your workflow to make use of them, if that's even possible. Most programmers wouldn't know how to do that/wouldn't care to bother. It's therefore reasonable to assume that a 5x/10x greater output, if it exists, is unevenly distributed, mostly affecting power users/ people particularly talented at using LLMs. Empirically, we likewise don't seem to be living in the world where the whole software industry is suddenly 5-10 times more productive. It'll have been the case for 1-2 years now, and I, at least, have felt approximately zero impact. I don't see 5-10x more useful features in the software I use, or 5-10x more software that's useful to me, or that the software I'm using is suddenly working 5-10x better, etc. However, I'm also struggling to see the supposed 5-10x'ing anywhere else. If power users are experiencing this much improvement, what projects were enabled by it? Previously, I'd assumed I didn't know just because I'm living under a rock. So I've tried to get Deep Research to fetch me an overview, and it... also struggled to find anything concrete. Judge for yourself: one, two. The COBOL refactor counts, but that's about it. (Maybe I'm bad at prompting it?) Even the AGI labs' customer-facing offerings aren't an endless trove of rich features for interfacing with their LLMs in sophisticated ways - even though you'd assume there'd be an unusual concentration of power users there. You have a dialogue box and can upload PDFs to it, that's about it. You can't get the LLM to interface with an ever-growing list of arbitrary software and data types, there isn't an endless list of QoL features that you can turn on/off on demand, etc.^[1] So I'm asking LW now: What's the real-world impact? What projects/ advancements exist now that wouldn't have existed without LLMs? And if none of that is publicly attributed to LLMs, what projects have appeared suspiciously fast, such that, on sober analysis, they couldn't have been spun up this quickly in the dark pre-LLM ages? What slice through the programming ecosystem is experiencing 10x growth, if any? And if we assume that this is going to proliferate, with all programmers attaining the same productivity boost as the early adopters are experiencing now, what would be the real-world impact? To clarify, what I'm not asking for is: * Reports full of vague hype about 10x'ing productivity, with no clear attribution regarding what project this 10x'd productivity enabled. (Twitter is full of those, but light on useful stuff actually being shipped.) * Abstract economic indicators that suggest X% productivity gains. (This could mean anything, including an LLM-based bubble.) * Abstract indicators to the tune of "this analysis shows Y% more code has been produced in the last quarter". (This can just indicate AI producing code slop/bloat). * Abstract economic indicators that suggest Z% of developers have been laid off/junior devs can't find work anymore. (Which may be mostly a return to the pre-COVID normal trends.) * Useless toy examples like "I used ChatGPT to generate the 1000th clone of Snake/of this website!". * New tools/functions that are LLM wrappers, as opposed to being created via LLM help. (I'm not looking for LLMs-as-a-service, I'm looking for "mundane" outputs that were produced much faster/ better due to LLM help.) I. e.: I want concrete, important real-life consequences. From the fact that I've observed none of them so far, and in the spirit of Cunningham's Law, here's a tentative conspiracy theory: LLMs mostly do not actually boost programmer productivity on net. Instead: * N hours that a programmer saves by generating code via an LLM are then re-wasted fixing/untangling that code. * At a macro-scale, this sometimes leads to "climbing up where you can't get down", where you use an LLM to generate a massive codebase, then it gets confused once a size/complexity threshold is passed, and then you have to start from scratch because the LLM made atrocious/alien architectural decisions. This likewise destroys (almost?) all apparent productivity gains. * Inasmuch as LLMs actually do lead to people creating new software, it's mostly one-off trinkets/proofs of concept that nobody ends up using and which didn't need to exist. But it still "feels" like your productivity has skyrocketed. * Inasmuch as LLMs actually do increase the amount of code that goes into useful applications, it mostly ends up spent on creating bloatware/services that don't need to exist. I. e., it actually makes the shipped software worse, because it's written more lazily. * People who experience LLMs improving their workflows are mostly fooled by the magical effect of asking an LLM to do something in natural language and then immediately getting kinda-working code in response. They fail to track how much they spend integrating and fixing this code, and/or how much the code is actually used. I don't fully believe this conspiracy theory, it feels like it can't possibly be true. But it suddenly seems very compelling. I expect LLMs have definitely been useful for writing minor features or for getting the people inexperienced with programming/with a specific library/with a specific codebase get started easier and learn faster. They've been useful for me in those capacities. But it's probably like a 10-30% overall boost, plus flat cost reductions for starting in new domains and for some rare one-off projects like " do a trivial refactor". And this is mostly where it'll stay unless AGI labs actually crack long-horizon agency/innovations; i. e., basically until genuine AGI is actually there. Prove me wrong, I guess. 1. ^^ Just as some concrete examples: Anthropic took ages to add LaTeX support, and why weren't RL-less Deep Research clones offered as a default option by literally everyone 1.5 years ago? ProgrammingSoftware ToolsAI Frontpage 116 Mentioned in 237A Bear Case: My Predictions Regarding AI Progress How Much Are LLMs Actually Boosting Real-World Programmer Productivity? 28Davidmanheim 18Thane Ruthenis 11Mis-Understandings 4Davidmanheim 4Garrett Baker 5Davidmanheim 27Richard Horvath 23faul_sname 3RussellThor 11chezdiogenes 13exmateriae 11Noosphere89 11Viliam 10yo-cuddles 10Raemon 10wachichornia 10CBiddulph 5David James 2Thane Ruthenis 9RobertM 8Archimedes 6azergante 6Michael Roe 5Gunnar_Zarncke 4Qumeric 4Yair Halberstadt 4Gunnar_Zarncke 3Jodie themathgenius 3jamii 3Ben Livengood 1Cole Wyeth 15Tao Lin 5Aprillion 11Daniel Kokotajlo 6Ajeya Cotra 9laserfiche New Answer New Comment Submit 19 Answers sorted by top scoring Davidmanheim Mar 04, 2025* 28 2 My understanding of the situations, speaking to people in normal firms who code, and management, is that this is all about theory of constraints. As a simplified example, if you previously needed 1 business analyst, one QA tester, and one programmer a day each to do a task, and the programmer's efficiency doubles, or quintuples, the impact on output is zero, because the firm isn't set up to go much faster. Firms need to rebuild their processes around this to take advantage, and that's only starting to happen, and only at some firms. Add Comment[lightbulb]1 [-]Thane Ruthenis4d18 4 That makes perfect sense to me. However, shouldn't we expect immediate impact on startups led by early adopters, which could be designed from the ground up around exploiting LLMs? Similar with AGI labs: you'd expect them to rapidly reform around that. Yet, I haven't heard of a single example of a startup shipping a five-year/ten-year project in one year, or a new software firm eating the lunch of ossified firms ten times its size. Reply [-]Mis-Understandings4d11 0 There is no reorganization that can increase the tempo of other organizations (pace of customer feedback), which is often the key bottleneck in software already. The same speed dynamic is not new, it is just in sharper focus. Reply 4Davidmanheim4d Lemonade is doing something like what you describe in Insurance. I suspect other examples exist. But most market segments, even in "pure" software, don't revolve around only the software product, so it is slower to become obvious if better products emerge. [-]Garrett Baker3d4 1 Why wouldn't you see the firm freeze programmer hires and start laying off people en mass? Reply 5Davidmanheim3d I'm not sure we'd see this starkly if people can change roles and shift between job types, but haven't we seen firms engage in large rounds of layoffs and follow up by not hiring as many coders already over the past couple years? Richard Horvath Mar 05, 2025 27 18 In my personal experience, LLMs can speed up me 10X times only in very specific circumstances, that are (and have always been) a minor part of my job, and I suspect this is true for most developers: * If I need some small, self-contained script to do something relatively simple. E.g. a bash script for file manipulation or an excel VBA code. These are easy to describe just with a couple of sentences, easy to verify, I often don't have deep enough knowledge in the particular language to write the whole thing without looking up syntax. And more importantly: as they are small and self-contained, there is no actual need to think much about the maintainability, such as unit testing, breaking into modules and knowing business and most environment context. (Anything containing regex is a subset of this, where LLMs are game changer) * If I need to work on some stack I am not too familiar with, LLMs can speed up learning process a lot by providing working (or almost working) examples. I just type what I need and I can get a rough solution that shows what class and methods I can use from which library, or even what general logic/approach can work as a solution. In a lot of cases, even if the solution provided does not actually work, just knowing about how I can interact with a library is a huge help. Without this I would have to spend hours or days either doing some online course/reading documentation/experimenting. The cavet here is that even in cases when the solution works, the provided code is not something that I can simply add into an actual project. Usually it has to be broken up and parts may go to different modules, refactored to be object oriented and use consistent abstractions that make sense for the particular project. Based on above, I suspect that most people who report 10X increase in software development either: 1. Had very low levels of development knowledge, so the bar of 10X is very low. Likely they also do not recognize/not yet experienced that plugging raw LLM code into a project can have a lot of negative downstream consequences that you described. (Though to be fair, if they could not write it before, some code is still better than no code at all, so a worthy trade-off). 2. Need to write a lot of small, self-contained solutions (I can imagine someone doing a lot of excel automation or being a dedicated person to build the shell scripts for devops/operations purposes). 3. Need to experiment with a lot of different stacks/libraries (although it is difficult to imagine a job where this is the majority of long-term tasks) 4. Were affected by the anchoring effect: where they recently used LLMs to solve something quickly, and extrapolated from that particular example, not from long-term experience. Add Comment faul_sname Mar 04, 2025 23 3 Inasmuch as LLMs actually do lead to people creating new software, it's mostly one-off trinkets/proofs of concept that nobody ends up using and which didn't need to exist. But it still "feels" like your productivity has skyrocketed. I've personally found that the task of "build UI mocks for the stakeholders", which was previously ~10% of the time I spent on my job, has gotten probably 5x faster with LLMs. That said, the amount of time I spend doing that part of my job hasn't really gone down, it's just that the UI mocks are now a lot more detailed and interactive and go through more iterations, which IMO leads to considerably better products. "This code will be thrown away" is not the same thing as "there is no benefit in causing this code to exist". The other notable area I've seen benefits is in finding answers to search-engine-proof questions - saying "I observe this error within task running on xyz stack, here is how I have kubernetes configured, what concrete steps can I take to debug such the system?" But it's probably like a 10-30% overall boost, plus flat cost reductions for starting in new domains and for some rare one-off projects like "do a trivial refactor". Sounds about right - "10-30% overall productivity boost, higher at the start of projects, lower for messy tangled legacy stuff" aligns with my observations, with the nuance that it's not that I am 10-30% more effective at all of my tasks, but rather that I am many times more effective at a few of my tasks and have no notable gains on most of them. And this is mostly where it'll stay unless AGI labs actually crack long-horizon agency/innovations; i. e., basically until genuine AGI is actually there. FWIW I think the bottleneck here is mostly context management rather than agency. Add Comment[bullseye]3[exclamatio]1 [-]RussellThor4d3 0 Yes to much of this. For small tasks or where I don't have specialist knowledge I can get 10* speed increase - on average I would put 20%. Smart autocomplete like Cursor is undoubtably a speedup with no apparent downside. The LLM is still especially weak where I am doing data science or algorithm type work where you need to plot the results and look at the graph to know if you are making progress. Reply chezdiogenes Mar 04, 2025 11 1 I'll offer a few cents which is probably what my input is literally worth as a layman, but I'm now able to build full-stack apps, soup to nuts, as someone who's entire coding experience to date was a helloworld in a python tutorial when I was a child. Cursor/Claude et al can now one shot entire single purpose apps, and errors seem to be on a downward trend. The N hours lost untangling code don't take into account the time saved in construction; but keeping that in mind how much time is spent in debug? What's the saying? 50% of time spent in programming is coding and the 90% is debugging? I know that's a joke but now you've largely wiped out that 50% and that 90% is chopped in half and is getting lower. Therein lies my point. There's no evidence of massive 5-10x advancements right now, but that's because of temporary roadblocks that will be iterated out in time. And we aren't talking decades of development any more. What's going to happen when no-code apps can churn out scads of clean, seamless structured code? No bugs? What then? Add Comment [-]exmateriae4d13 0 To add to this, I'm a forecaster on metaculus and I can now do dozens if not hundred of poisson/monte carlo/ets simulations every hour when before I often needed the hour to do two or three because I had to do small tweaks that took me quite some time before and that I now delegate to AI. I learned python a year ago but clearly I'm a newbie, it has changed my capabilities significantly. Reply Noosphere89 Mar 04, 2025 11 4 Re AI coding, some interesting thoughts on this are from Ajeya Cotra's talks (short form, there are a lot of weaknesses, but the real-world programmer productivity is surprisingly high for coding tasks, but is very bad outside of coding tasks, which is why AI's impact is limited so far): https://x.com/ajeya_cotra/status/1894821432854749456 https://x.com/ajeya_cotra/status/1895161774376436147 Re this: And this is mostly where it'll stay unless AGI labs actually crack long-horizon agency/innovations; i. e., basically until genuine AGI is actually there. Prove me wrong, I guess. My main takeaway is actually kind of different, in that there's less of a core of generality on the default path than people originally thought, and while there is some entanglement on capabilities, there can also be weird spikes and deficits, and this makes the term AGI a lot less useful than people thought. Add Comment [-]Viliam2d11 0 The AI does not make the meetings pass 10x faster, and that is where the senior developers spend a lot of time. Reply yo-cuddles Mar 05, 2025* 10 13 This is not going to be a high quality answer, sorry in advance. I noticed this with someone in my office who is learning robotic process automation: people are very bad at measuring their productivity, they are better at seeing certain kinds of gains and certain kinds of losses. I know someone who swears emphatically that they are many times as productive but have become almost totally unreliable. He's in denial over it, and a couple people now have openly told me they try to remove him from workflows for all the problems he causes. I think the situation is like this: If you finish a task very quickly using automated methods, that feels viscerally great and, importantly, is very visible. If your work then incurs time costs later, you might not be able to trace that extra cost to the "automated" tasks you set up earlier, double so if those costs are absorbed by other people catching what you missed and correcting your mistakes, or doing the things that used to be done when you were doing it manually. I imagine it is hard to track a bug and know, for certain, that you had to waste that time because you used an LLM instead of just doing it yourself. You don't know who else had to waste time fixing your problem because LLM code is spaghetti, or at least you don't feel it in your bones in the same way you feel increases in your output, you don't get to see the counterfactual project where things just went better in intangible ways because you didn't outsource your thinking to gpt. Few people notice, after the fact, how many problems they incurred because of a specific thing they did. I think LLM usage is almost ubiquitous at this point, if it were conveying big benefits it would show more clearly. If everyone is saying they are 2x more productive (which is kinda low by some testimonies) then it is probably the case that they are just oblivious to the problems they are causing for themselves because they're just less visible. Add Comment Raemon Mar 05, 2025 10 3 One thing is I'm definitely able to spin up side projects that I just would not have been able to do before, because I can do them with my "tired brain." Some of them might turn out to be real projects, although it's still early stage. Add Comment wachichornia Mar 04, 2025 10 2 My current job is to develop PoCs and iterate over user feedback. It is a lot of basic and boilerplate. I am handling three projects at the same time when before cursor I would have been managing one and taking longer. I suck at UI and cursor simply solved this for me. We have shipped one of the tools and are finalizing shipping the second one, but they are LLM wrappers indeed designed to summarize or analyze text for customer support purposes and GTR related stuff. The UI iteration however has immensely helped and accelerated. Add Comment CBiddulph Mar 04, 2025 10 3 Yeah, 5x or 10x productivity gains from AI for any one developer seem pretty high, and maybe implausible in most cases. However, note that if 10 people in a 1,000-person company get a 10x speedup, that's only a ~10% overall speedup, which is significant but not enough that you'd expect to be able to clearly point at the company's output and say "wow, they clearly sped up because of AI." For me, I'd say a lot of my gains come from asking AI questions rather than generating code directly. Generating code is also useful though, especially for small snippets like regex. At Google, we have something similar to Copilot that autocompletes code, which is one of the most useful AI features IMO since the generated code is always small enough to understand. 25% of code at Google is now generated by AI, a statistic which probably mostly comes from that feature. There are a few small PRs I wrote for my job which were probably 5-10x faster than the counterfactual where I couldn't use AI, where I pretty much had the AI write the whole thing and edited from there. But these were one-offs where I wasn't very familiar with the language (SQL, HTML), which means it would have been unusually slow without AI. Add Comment [-]David James4d5 0 For me, I'd say a lot of my gains come from asking AI questions rather than generating code directly. This is often the case for me as well. I often work on solo side projects and use Claude to think out loud. This lets me put on different hats, just like when pair programming, including: design mode, implementation mode, testing mode, and documentation mode. I rarely use generated code as-is, but I do find it interesting to look at. As a concrete example, I recently implemented a game engine for the board game Azul (and multithreaded solver engine) in Ru... (read more) Reply [-]Thane Ruthenis4d2 0 However, note that if 10 people in a 1,000-person company get a 10x speedup, that's only a ~10% overall speedup Plausible. Potential counter-argument: software engineers aren't equivalent, "10x engineers" are a thing, and if we assume they're high-agency people always looking to streamline and improve their workflows, we should expect them to be precisely the people who get a further 10x boost from LLMs. Have you observed any specific people suddenly becoming 10x more prolific? But these were one-offs where I wasn't very familiar with the language (SQL, HTML ... (read more) Reply 9RobertM4d In addition to the objection from Archimedes, another reason this is unlikely to be true is that 10x coders are often much more productive than other engineers because they've heavily optimized around solving for specific problems or skills that other engineers are bottlenecked by, and most of those optimizations don't readily admit of having an LLM suddenly inserted into the loop. 8Archimedes4d I highly doubt this. A 10x engineer is likely already bottlenecked by non-coding work that AI can't help with, so even if they 10x their coding, they may not increase overall productivity much. azergante Mar 05, 2025 6 3 Many developers have been reporting that this is dramatically increasing their productivity, up to 5x'ing/10x'ing it I challenge the data: none of my colleagues have been reporting this high a speed-up. I think your observation can just be explained by a high sampling bias. People who do not use AI or got no improvement are unlikely to report. You also mention Twitter where users share "hot takes" etc to increase engagement. It's good to have actual numbers before we explain them, so I ran a quick search and found 3 articles that look promising (I only did a basic check on the methodology, don't take these numbers at face value without analyzing the source in depth): * An Axify analysis of the DORA metrics they collect: https:// axify.io/blog/use-ai-for-developer-productivity Documentation quality (+7.5%) Code review speed (+3.1%) Delivery throughput (-1.5%): AI adoption slightly decreases delivery throughput, usually due to over-reliance, learning curve, and increased complexity. Delivery stability (-7.2%): It is significantly impacted because AI tools can generate incorrect or incomplete code, increasing the risk of production errors. What are the DORA metrics? Deployment frequency | How often a team puts an item into production. Lead time for changes | Time required for a commit to go into production. Change failure rate | Percentage of deployments resulting in production failure. Failed deployment recovery time | Time required for a team to recover from a production failure. * A McKinsey pilot study on 40 of their developers: https:// www.mckinsey.com/capabilities/mckinsey-digital/our-insights/ unleashing-developer-productivity-with-generative-ai [c8zab3omse] * An UpLevel analysis of the data of 800 developers: https:// uplevelteam.com/blog/ai-for-developer-productivity Analyzing actual engineering data from a sample of nearly 800 developers and objective metrics, such as cycle time, PR throughput, bug rate, and extended working hours ("Always On" time), we found that Copilot access provided no significant change in efficiency metrics. The group using Copilot introduced 41% more bugs Copilot access didn't mitigate the risk of burnout "The adoption rate is significantly below 100% in all three experiments," the researchers wrote. "With around 30-40% of the engineers not even trying the product. In my experience LLMs are a replacement for search engines (these days, search engines are only good to find info when you already know on which website to look ...). They don't do well in moderately sized code bases, nor in code bases with lots of esoteric business logic, which is to say they don't do well in most enterprise software. I mostly use them as: * a documentation that talks: it reminds me of syntax, finds functions in APIs and generates code snippets I can play with to understand APIs * a fuzzy version of an encyclopedia: it helps me get an overview of a subject and points me to resources where I can get crisper knowledge * support/better search engine for weird build issues, or weird bugs etc I think it's also good at one shot scripts, such as data wrangling and data viz, but it does not come up often in my current position. I would rate the productivity increase at about 10%. I think the use of modal editors (Vim or modern alternatives) improve coding speed more than inline AI completion, which is often distracting. A lot of my time is spent understanding what I need to code in a back and forth with the product manager or clients. Then I can either code it myself (it's not so hard once I understand the requirements) or spend time explaining the AI what it needs to do, watch it write sloppy code, and rewrite the thing. Once in a while the coding part is actually the hard part, for example when I need to make the code fast or make a feature play well with the existing architecture, but then the AI can't do logic well enough to optimize code, nor can it reason about the entire code base which it can't see anyway. Add Comment[check]1 Michael Roe Mar 05, 2025 6 0 One possible explanation is the part of the job that gets speeded up by LLMs is a relatively small part of what programmers actually do, so the total speed up is small. What programmers do includes: Figuring out what the requirements are - this might involve talking to who the software is being produced for Writing specifications Writing tests Having arguments discussion during code review when the person reviewing your code doesn't agree with the way you did it Etc. etc. Personally, I find that LLMs are nearly there, but not good enough just yet. Add Comment Gunnar_Zarncke Mar 06, 2025 5 0 Data point: For 25% of the [Y Combinator] Winter 2025 batch, 95% of lines of code are LLM generated. Add Comment Qumeric Mar 06, 2025 4 0 I am confident that LLMs siginificantly boost software development productivity (I would say 20-50%) and am completely sure it's not even close to 5x. However, despite I agree with your conclusion, I would like to point out that timeframes are pretty short. 2 years ago (~exactly GPT-4 launch date) LLMs were barely making any impact. I think tools started to resemble the current state around 1 year ago (~exactly Claude 3 Opus launch date). Now, suppose we had 5x boost for a year. Would it be very visible? We would have got 5 years of progress in 1 year but had software landscape changed a lot in 5 years in pre-LLM era? Comparing 2017 and 2022, I don't feel like that much changed. Add Comment Yair Halberstadt Mar 06, 2025 4 2 LLMs make certain tasks much faster. Those tasks were never a large part of my job, and so whilst they might increase my productivity 10X on one specific thing, my overall productivity is probably up about 5%. LLMs excel at: 1. Writing quick scripts to do some specific task. 2. Greenfield projects I don't need to maintain. LLMs help at: 1. Researching specific libraries or standard solutions to something (e.g. how do I format a number to two decimal places in go). 2. Predicting my next 10 lines of code, (which I then need to go and fix up afterwards). LLMs suck at: 1. Making changes to a large codebase. 2. Predicting what I'm going to write in a design document. So I'm now spending relatively less of my time fiddling with bash, and more on making changes to large codebases or writing design documents. Add Comment Gunnar_Zarncke Mar 05, 2025 4 0 The integration of GitHub's Copilot, into software development has been gradual. In June 2022, Copilot generated 27% of the code for developers using it, which increased to 46% by February 2023. As of August 2024, GitHub Copilot has approximately 2 million paying subscribers and is utilized by over 77,000 organizations. An increase from earlier figures; in February 2024, Copilot had over 1.3 million paid subscribers, with more than 50,000 organizations using the service. https://www.ciodive.com/news/ generative-AI-software-development-lifecycle/693165/ The transition from traditional waterfall methodologies to agile practices began in the early 2000s, with agile approaches becoming mainstream over the next decade. Similarly, the shift from monolithic architectures to microservices started around 2011, with widespread adoption occurring over the subsequent 5 to 10 years. https://en.wikipedia.org/wiki/Agile_software_development Assuming AI tools taking 5 years to become commonplace and leading to consistent 3x productivity gain (to use a more conservative number than the 5x/10x, but still high) this would lead to a nominal productivity gain of 25% per year. Significant, but not that trivial to clearly see. Add Comment Jodie themathgenius Mar 06, 2025 3 0 This is only my personal experience of course, but for making little tools for automating tasks or prototyping ideas quickly AI has been a game-changer. For web development the difference is also huge. The biggest improvement is when I'm trying something completely new to me. It's possible to confidently go in blind and get something done quickly. When it comes to serious engineering of novel algorithms it's completely useless, but most devs don't spend their days doing that. I've also found that getting a software job is much harder lately. The bar is extremely high. This could be explained by AI, but it could also be that the supply of devs has increased. Add Comment jamii Mar 06, 2025 3 0 There have been massive improvements in productivity in infrastructure from cloud services, especially object storage. See eg https://materializedview.io/p/infrastructure-vendors-are-in-a-tough for many examples of small teams out-competing established vendors. But I didn't notice that until it was pointed out to me by several sources. So should I expect to notice large productivity improvements from LLMs? How huge would they have to be to show up in the sparse signal that I'm getting? What data would I even have to be looking at to see a 5x improvement in productivity? Add Comment Ben Livengood Mar 04, 2025 3 0 For Golang: Writing unit test cases is almost automatic (Claude 3.5 and now Claude 3.7). It's good at the specific test setup necessary and the odd syntax/boilerplate that some testing libraries require. At least 5x speedup. Autosuggestions (Cursor) are net positive, probably 2x-5x speedup depending on how familiar I am with the codebase. Add Comment Cole Wyeth Mar 04, 2025 1 0 As you probably know, I have been endorsing this "conspiracy theory" for some time, e.g. roughly here: https://www.lesswrong.com/posts/ vvgND6aLjuDR6QzDF/my-model-of-what-is-going-on-with-llms Add Comment 5 comments, sorted by top scoring Click to highlight new comments since: Today at 11:00 PM [-]Tao Lin4d15 6 Empirically, we likewise don't seem to be living in the world where the whole software industry is suddenly 5-10 times more productive. It'll have been the case for 1-2 years now, and I, at least, have felt approximately zero impact. I don't see 5-10x more useful features in the software I use, or 5-10x more software that's useful to me, or that the software I'm using is suddenly working 5-10x better, etc. Diminishing returns! Scaling laws! One concrete version of "5x productivity" is "as much productivity as 5 copies of me in parallel", and we know that usually 5x-ing most inputs, like training compute and data, # of employees, etc, more often scales logarithmically instead of linearly Reply[lightbulb]1 [-]Aprillion3d5 2 that's not how productivity ought to be measured - it should measure some output per (say) a workday 1 vs 5 FTE is a difference in input, not output, so you can say "adding 5 people to this project will decrease productivity by 70% next month and we hope it will increase productivity by 2x in the long term" ... not a synonym of "5x productivity" at all it's the measure by which you can quantify diminishig results, not obfuscate them! ...but the usage of "5-10x productivity" seems to point to a diffent concept than a ratio of useful output per input AFAICT it's a synonym with "I feel 5-10x better when I write code which I wouldn't enjoy writing otherwise" Reply [-]Daniel Kokotajlo3d11 4 Many developers have been reporting that this is dramatically increasing their productivity, up to 5x'ing/10x'ing it. Yeah but many more have been saying it only moderately increases their productivity. Whenever I ask people working in frontier AI companies I get much smaller estimates, like 5%, 10%, 30%, etc. And I think @Ajeya Cotra got similar numbers and posted about them recently. Reply [-]Ajeya Cotra2d6 0 Yeah I've cataloged some of that here: https://x.com/ajeya_cotra/ status/1894821255804788876 Hoping to do something more systematic soon Reply [-]laserfiche2d9 -1 This is mostly covered by other comments, but I'll give my personal experience. I'm a programmer, and I have seen at least a 5x increase in programming speed. However, my output is limited by Amdahl's law. 50% of my process is coding, but 50% is in non-programming tasks that haven't been accelerated, meaning the overall process take 60% as long as originally. What's more, that's just my portion. When I complete a project and hand it off, the executives can't absorb it any faster, and my next assignment doesn't come any faster. My team doesn't work from an infinite backlog. Notably, my code is much higher quality (no more "eh, I'll get to that eventually") and always exhibits best practices and proper documentation, since those elements are now free. To take another example, I have used my accelerated programming speed to create several side project websites recently. Every part of this process has been accelerated, except for non-programming elements like spreading the word. That area is a bottleneck, and that's just my portion; even if I were keeping up with "marketing", there is a limit to how fast other parts of the information pipeline could absorb this new information and disseminate it, and how long adoption would take. Reply Moderation Log More from Thane Ruthenis View more Curated and popular this week A 195