[HN Gopher] Effects of Gen AI on High Skilled Work: Experiments ...
___________________________________________________________________
Effects of Gen AI on High Skilled Work: Experiments with Software
Developers
Author : Anon84
Score : 122 points
Date : 2024-09-06 11:21 UTC (11 hours ago)
(HTM) web link (papers.ssrn.com)
(TXT) w3m dump (papers.ssrn.com)
| danielvaughn wrote:
| A 26% productivity increase sounds inline with in my experience.
| I think one dimension they should explore is whether you're
| working with a new technology or one that you're already familiar
| with. AI helps me much more with languages/frameworks that I'm
| trying to learn.
| itchyjunk wrote:
| Maybe that's what the new vs experienced difference is
| indirectly capturing.
| jstummbillig wrote:
| That might be. It might also be a consequence of how flexible
| people at different stages in their career are.
|
| I have seen mostly senior programmers argue why ai tools don't
| work. Juniors just use them without prejudice.
| tuyiown wrote:
| Flexible, or just that only juniors have real benefits
| infecto wrote:
| I wish I could upvote this comment more than once. There does
| appear to be a prejudice with more senior programmers arguing
| why it cannot work, how they just cause more trouble, and
| other various complaints. The tools today are not perfect but
| they still amaze me at what is being accomplished, even a 10%
| gain is incredible for something that costs $10/month. I
| believe progress will be made in the space and the tooling in
| 5 years will be even better.
| randomdata wrote:
| _> The tools today are not perfect_
|
| The trouble is that they seem to be getting worse. Some
| time ago I was able to write an entire small application by
| simply providing some guidance around function names and
| data structures, with an LLM filling in all of the rest of
| the code. It worked fantastically and really showed how
| these tools can be a boon.
|
| I want to taste that same thrill again, but these days I'm
| lucky if I can get something out of it that will even
| compile, never mind the logical correctness. Maybe I'm just
| getting worse at using the tools.
| mattgreenrocks wrote:
| The prejudice comes down to whether they want to talk the
| LLM into the right solution vs applying what they know
| already. If you know your way around then there's no need
| to go through the LLM. I think sr devs often tend to be
| more task focused so the idea of outsourcing the thinking
| to an LLM feels like another step to take on.
|
| I find Claude good at helping me find how to do things that
| I know are possible but I don't have the right nomenclature
| for. This is an area where Google fails you, as you're
| hoping someone else on the internet used similar terms as
| you when describing the problem. Once it spits out some
| sort of jargon I can latch onto, then I can Google and find
| docs to help. I prefer to use multiple sources vs just
| LLMs, partially because of hallucination, but also to keep
| amassing my own personal context. LLMs are excellent as
| librarians.
| jncfhnb wrote:
| My experience is juniors using them without prejudice and
| then not understanding why their code is wrong
| SketchySeaBeast wrote:
| I am seeing that a lot - juniors who can put out a lot of
| code but when they get stuck they can't unstick themselves,
| and it's hard for me to unstick them because they have a
| hard time walking me through what they are doing.
| carlmr wrote:
| I've gotten responses now on PRs of the form. "I don't
| know either, this is what Copilot told me."
|
| If you don't even understand your own PR, I'm not sure
| why you expect other people can.
|
| I have used LLMs myself, but mostly for boilerplate and
| one-off stuff. I think it can be quite helpful. But as
| soon as you stop understanding the code it generates you
| will create subtle bugs everywhere that will cost you
| dearly in the long run.
|
| I have the strong feeling that if LLMs really outsmart us
| to the degree that some AI gung-ho types believe, the old
| Kernighan quote will get a new meaning:
|
| "Debugging is twice as hard as writing the code in the
| first place. Therefore, if you write the code as cleverly
| as possible, you are, by definition, not smart enough to
| debug it."
|
| We'll be left with code nobody can debug because it was
| handed to us by our super smart AI that only hallucinates
| sometimes. We'll take the words of another AI that the
| code works. And then we'll hope for the best.
| SketchySeaBeast wrote:
| Copilot is best for me when it's letting me hit tab to
| auto-complete something I was going to write anyways.
| ebiester wrote:
| Is that so new? We used to complain when someone blindly
| copied and pasted from stack overflow. Before that, from
| experts-exchange.
|
| Coding is still a skill acquisition that takes years. We
| need to stamp out the behavior of not understanding what
| they take from copilot, but the behavior is not new.
| SketchySeaBeast wrote:
| You're right, it isn't entirely new, but I think it's
| still different. You still had to figure out how that new
| code snippet applied to your code. Stack overflow
| wouldn't shape the code you're trying to slot in to fit
| your current code.
| GoToRO wrote:
| As a senior, you know the problem is actually finishing a
| project. That's the moment all those bad decisions made by
| junior need to be fixed. This also means that a 80% done
| project is more like 20% done, because in the state it is, it
| can not be finished: you fix one thing and break 2 more.
| SketchySeaBeast wrote:
| As a senior, I find that trying to use copilot really only
| gives me gains maybe half the time, the other half the time
| it leads me in the wrong direction. Googling tends to give me
| a better result because I can actually move through the data
| quicker. My belief is this is because when I need help I'm
| doing something uncommon or hard, as opposed to juniors who
| need help doing regular stuff which will have plenty of
| examples in the training data. I don't need help with that.
|
| It certainly has its uses - it's awesome at mocking and
| filling in the boilerplate unit tests.
| marginalia_nu wrote:
| I find their value depends a lot on what I'm doing. Anything
| easy I'll get insane leverage, no exaggeration I'll slap
| together that shit 25x faster. It's seen likely billions of
| lines of simple CRUD endpoints, so yeah it'll write those
| flawlessly for you.
|
| Anything the difficult or complex, and it's really a coinflip
| if it's even an advantage, most of the time it's just
| distracting and giving irrelevant suggestions or bad
| textbook-style implementations intended to demonstrate a
| principle but with god-awful performance. Likely because
| there's simply not enough training data for these types of
| tasks.
|
| With this in mind, I don't think it's strange that junior
| devs would be gushing over this and senior devs would be
| raising a skeptical eyebrow. Both may be correct, depending
| on what you work on.
| jeremy151 wrote:
| I think for me, I'm still learning how to make these tools
| operate effectively. But even only a few months in, it has
| removed most all the annoying work and lets me concentrate
| on the stuff that I like. At this point, I'll often give it
| some context, tell it what to make and it spits out
| something relatively close. I look it over, call out like
| 10 things, each time it says "you're right to question..."
| and we do an iteration. After we're thru that, I tell it to
| write a comprehensive set of unit tests, it does that, most
| of them fail, it fixes them, and them we usually have
| something pretty solid. Once we have that base pattern, I
| can have it pattern and extend variants after the first
| solid bit of code. "Using this pattern for style and
| approach, make one that does XYZ instead."
|
| But what I really appreciate is, I don't have to do the
| plug and chug stuff. Those patterns are well defined, I'm
| more than happy to let the LLM do that and concentrate on
| steering whether it's making a wise conceptual or
| architectural choice. It really seems to act like a higher
| abstraction layer. But I think how the engineer uses the
| tool matters too.
| SoftTalker wrote:
| Juniors don't know enough to know what problems the AI code
| might be introducing. It might work, and the tests might
| pass, but it might be very fragile, full of duplicated code,
| unnecessary side-effects, etc. that will make future
| maintenance and debugging difficult. But I guess we'll be
| using AI for that too, so the hopefully the AI can clean up
| the messes that it made.
| nuancebydefault wrote:
| Now some junior dev can quickly make something new and fully
| functional in days, without knowing in detail what they are
| doing. As opposed to weeks by a senior originally.
|
| Personally I think that senior devs might fear a conflict
| within their identity. Hence they draw the 'You and the AI
| have no cue' card.
| pqdbr wrote:
| Same impression here.
|
| I've been using Cursor for around 10 days on a massive Ruby on
| Rails project (a stack I've been coding in for +13 years).
|
| I didn't enjoy any productivity boost on top of what GitHub
| Copilot already gave me (which I'd estimate around the 25%
| mark).
|
| However, for crafting a new project from scratch (empty folder)
| in, say, Node.js, it's uncanny; I can get an API serving
| requests from a OpenAPI schema (serving the OpenAPI schema via
| swagger) in ~5 minutes just by prompting.
|
| Starting a project from scratch, for me at least, is rare,
| which probably means going back to Copilot and vanilla VSCode.
| gamerDude wrote:
| I start new projects a lot and just have a template that has
| everything I would need already set up for me. Do you think
| there is a unique value prop that AI gives you when setting
| up that a template would not have?
| carlmr wrote:
| AI has templates for the languages you seldom use (as long
| as the language is not too niche for the LLM to have
| trained on it).
| lherron wrote:
| I do both. I have a basic template that also includes my
| specific AI instructions and conventions that are inputs
| for Aider/Cursor. Best of both worlds.
| pjot wrote:
| GH Copilot chat specifically has directives like
| @workspace /new "scaffold a ${language} project"
|
| that automagically creates a full project structure and
| boilerplate. It's been great for one off things for me at
| least
| sethops1 wrote:
| You don't need AI to implement that.
| CuriouslyC wrote:
| To fully extract maximum value out of LLMs, you need to
| change how you architect and organize software to make it
| easy for them to work with. LLMs are great with function
| libraries and domain specific languages, so the more of your
| code you can factor into those sorts of things the greater a
| speed boost they'll give you.
| dartos wrote:
| How do you use AI in your workflow outside of copilot?
|
| I haven't been able to get any mileage out of chat AI beyond
| treating it like a search engine, then verifying what it
| said.... Which isn't a speedy workflow
| SketchySeaBeast wrote:
| I feel like this whole "starting a new project" might be the
| divide between the jaded and excited, which often (but not
| always) falls between senior and junior lines. I just don't
| do that anymore. Coding is no longer my passion, it's my
| profession. I'm working in an established code base that I
| need to thoughtfully expand and improve. The easy and
| boilerplate problems are solved. I can't remember the last
| time I started up a new project, so I never see that side of
| copilot or cursor. Copilot might at its best when tinkering.
| yunwal wrote:
| If I struggle on a particularly hard implementation detail in
| a large project, often I'll use an LLM to set up a
| boilerplate version of the project from scratch with fewer
| complications so that I can figure out the problem. It gets
| confused if it's interacting with too many things, but the
| solutions it finds in a simple example can often be
| instructive.
| moolcool wrote:
| I've found Copilot pretty good at removing some tedium, like
| it'll write docstrings pretty well most of the time, but it
| does almost nothing to alleviate the actual mental labour of
| software engineering
| jacobsenscott wrote:
| Removing the tedium frees your brain up to focus on the
| interesting parts of the problem. The "actual mental labor"
| is the fun part. So I like that.
| danielvaughn wrote:
| Yeah this has been my experience. I bucket my work largely
| into two categories - "creative work" and "toil". I don't
| want or need AI to replace my creative work, but the more
| it can handle toil for me, the better.
|
| As it happens, I think co-pilot is a pretty poor user
| experience, because it's essentially just an autocomplete,
| which doesn't really help me all that much, and often gets
| in my way. I like using Cursor with the autocomplete turned
| off. It gives you the option to highlight a bit of text and
| either refactor it with a prompt, or ask a question about
| it in a side chat window. That puts me in the driver seat,
| so to speak, so I (the user) can reach out to AI when I
| want to.
| hobofan wrote:
| I'd also expand it to "languages/frameworks that I'll never
| properly learn".
|
| I'm not great at remembering specific quirks/pitfalls about
| secondary languages like e.g. what the specific quoting
| incantations are to write conditionals in Bash, so I rarely
| wrote bash scripts for automation in the past. Basically only
| if that was a common enough task to be worth the effort. Same
| for processing JSON with jq, or parsing with AWK.
|
| Now with LLMs, I'm creating a lot more bash scripts, and it has
| gotten so easy that I'll do it for process-documentation more
| often. E.g. what previously was a more static step-by-step
| README with instructions is now accompanied with an interactive
| bash script that takes user input.
| sgarland wrote:
| While I'll grant you that LLMs are in fact shockingly good
| with shell scripts, I also highly recommend shellcheck [0].
| You can get it as a treesitter plugin for nvim, and I assume
| others, so it lints as you write.
|
| [0]: https://www.shellcheck.net/
| danielvaughn wrote:
| Oh god yes, LLM for bash scripting is fantastic. Bash is one
| of those languages that I simply don't want to learn. Reading
| it makes my eyes hurt.
| a_wild_dandan wrote:
| You could try zx[1]! I prefer it to Bash scripting.
|
| [1] https://github.com/google/zx
| einpoklum wrote:
| You could, but then your script won't run anywhere except
| your own system, as opposed to running basically
| everywhere...
| iwontberude wrote:
| Bash is a terrible language in so so many ways and I concur
| that I have no interest in learning its new features going
| forward. I remember how painfully inadequate it was for
| actual computation and its data types are non existent.
| Gross.
| SoftTalker wrote:
| You have to understand the domain in which its intended
| to be used.
|
| Bash scripts are essentially automating what you could do
| at the command line with utility programs, pipes,
| redirects, filters, and conditionals.
|
| If you're getting very far outside of that scope, bash is
| probably the wrong tool (though it can be bent to do just
| about anything if one is determined enough).
| danielvaughn wrote:
| I don't know too much about the history, but wasn't this
| one of the original motivations for Python? Like it was
| meant to be basically "bash scripting but with better
| ergonomics", or something like that.
| okwhateverdude wrote:
| You might thinking of Perl. Affectionately known as the
| swiss army chainsaw of programming languages, it
| incorporated elements from bash, awk, sed, etc and
| smushed them together into a language that carried the
| early Web.
| empath75 wrote:
| Sounds about right to me, which is why the hysteria about AI
| wiping out developer jobs was always absurd. _Every_ time there
| has been a technology that improved developer productivity,
| developer jobs and pay have _increased_. There is not a limited
| amount of automation that can be done in the world and the
| cheaper and easier it gets to automate stuff, the more stuff
| will be economically viable to automate that wasn't before. Did
| IDE's eliminate developer jobs? Compilers? It's just a tool.
| randomdata wrote:
| The automated elevator is just a tool, but it "wiped out" the
| elevator operator. Which is really to say not that the
| elevator operator was wiped out, but that _everyone_ became
| the elevator operator. Thus, by the transitive properties of
| supply and demand, the value of operating an elevator
| declined to nothing.
|
| Said hysteria was built on the same idea. After all, LLMs
| themselves are just compilers for a programming language that
| is incredibly similar to spoken language. But as the
| programming language is incredibly similar to the spoken
| language that nearly everyone already knows, the idea was
| that everyone would become the metaphorical elevator
| operator, "wiping out" programming as a job just as elevator
| operators were "wiped out" of a job when operating an
| elevator became accessible to all.
|
| The key difference, and where the hysteria is likely to fall
| flat, is that when riding in an elevator there isn't much
| else to do but be the elevator operator. You may as well do
| it. Your situation would not be meaningfully improved if
| another person was there to press the button for you. When it
| comes to programming, though, there is more effort involved.
| Even when a new programming language makes programming
| accessible, there remains a significant time commitment to
| carry out the work. The business people are still best to
| leave that work to the peons so they can continue to focus on
| the important things.
| insane_dreamer wrote:
| I haven't found it that useful in my main product development
| (which while python based, uses our own framework and therefore
| there's not much code for CoPilot to go on and it usually
| suggests methods and arguments that don't exist, which just
| makes extra work).
|
| Where I do find it useful are
|
| 1) questions about frameworks/languages that I don't work in
| much and for which there is a lot of example content (i.e., Qt,
| CSS);
|
| 2) very specific questions I would have done a Google Search
| (usually StackOverflow) for ("what's the most efficient way to
| CPU and RAM usage on Windows using python") - the result is
| pointing me to a library or some example rather than directly
| generating code that I can copy/paste
|
| 3) boilerplate code that I already know how to write but saves
| me a little time and avoids typing errors. I have the CoPilot
| plugin for PyCharm so I'll write it as a comment in the file
| and then it'll complete the next few lines. Again best results
| is something that is very short and specific. With anything
| longer I almost always have to iterate so much with CoPilot
| that it's not worth it anymore.
|
| 4) a quick way to search documentation
|
| Some people have said it's good at writing unit tests but I
| have not found that to be the case (at least not the right kind
| of unit tests).
|
| If I had to quantify it, I'd probably give it a 5-10% increase
| in productivity. Much less than I get from using a full
| featured IDE like PyCharm over coding in Notepad, or a really
| good git client over typing the git commands in the CLI. In
| other words, it's a productivity tool like many other tools,
| but I would not say it's "revolutionary".
| nuancebydefault wrote:
| It is revolutionary no doubt. How many pre-AI tools would
| accomplish an amount of 4 big features you mentioned at once?
| insane_dreamer wrote:
| Well, like I said, a well-designed IDE is a much bigger
| productivity booster than CoPilot, and I've never heard
| anyone describe JetBrains, NetBeans or VSCode as
| "revolutionary".
| nuancebydefault wrote:
| I would call IDE's "evolutionary". Jetbrains is a better
| evolutionary version of, say, Eclipse in the day.
| rm_-rf_slash wrote:
| AI speeds my coding up 2x-4x, depending on the breadth of
| requirements and complexity of new tech to implement.
|
| But coding is just a fraction of my weekly workload, and AI has
| been less impactful for other aspects of project management.
|
| So overall it's 25%-50% increase in productivity.
| jakub_g wrote:
| I only use (free) ChatGPT sporadically, and it works best for
| me in areas where I'm familiar enough to call bullshit, but not
| familiar enough to write things myself quickly / confidently /
| without checking a lot of docs:
|
| - writing robust bash and using unix/macos tools
|
| - how to do X in github actions
|
| - which API endpoint do I use to do Y
|
| - synthesizing knowledge on some topic that would require
| dozens of browser tabs
|
| - enumerating things to consider when investigating things.
| Like "I'm seeing X, what could be the cause, and how I do check
| if it's that". For example I told it last week "git rebase is
| very slow, what can it be?" and it told me to use GIT_TRACE=1
| which made me find a slow post-commit hook, and suggested how
| to skip this hook while rebasing.
| alternatex wrote:
| I wonder if the study includes the technical debt that more
| experienced developers had to tackle after the less experienced
| devs have contributed their AI-driven efforts. Because my
| personal experience has involved a lot of that in one of the
| companies listed in the study.
|
| Also, I've personally seen more interest in AI in devs that have
| little interest in technology, but a big interest in delivering.
| PMs love them though.
| anonzzzies wrote:
| > had to tackle after the less experienced devs have
| contributed their AI-driven efforts.
|
| So, like before AI then? I haven't seen AI deliver illogical
| nonsense that I couldn't even decipher like I have seen some
| outsourcing companies deliver.
| htrp wrote:
| AI seems to give more consistent results than outsourcing
| (not necessarily better, but at least more predictable
| failure modes)
| trashtester wrote:
| > that I couldn't even decipher
|
| This is one of the challenges of being a tech lead. Sometimes
| code is hard to comprehend.
|
| In my experience, AI delivered code is no worse than entry
| level developer code.
| sgarland wrote:
| > Also, I've personally seen more interest in AI in devs that
| have little interest in technology, but a big interest in
| delivering.
|
| Thank you, this says what I have been struggling to describe.
|
| The day I lost part of my soul was when I asked a dev if I
| could give them feedback on a DB schema, they said yes, and
| then cut me off a few minutes in with, "yeah, I don't really
| care [about X]." You don't care? I'm telling you as the SME for
| this exactly what can be improved, how to do it, and why you
| should do so, but you don't care. Cool.
|
| Cloud was a mistake; it's inculcated people with the idea that
| chasing efficiency and optimization doesn't matter, because you
| can always scale up or out. I'm not even talking about doing
| micro-benchmarks (though you should...), I'm talking about
| dead-simple stuff like "maybe use this data structure instead
| of that one."
| dartos wrote:
| I've noticed this more and more since 2020.
|
| A lot of people who entered the field in the past 6 or so
| years are here for the money, obviously.
|
| Nothing wrong with that at all, but as someone with a long
| time programming and technology passion, it's sad to see that
| change.
| sgarland wrote:
| There's certainly nothing wrong with enjoying the high pay,
| no - I definitely do. But yeah, it's upsetting to find out
| how few people care. Even moreso when they double down and
| say that you shouldn't care either, because it's been
| abstracted away, blah blah blah. Who do you think is going
| to continue creating these abstractions for you?
| ebiester wrote:
| ...Someone else who thinks of actual web applications as
| an abstraction.
|
| In the olden days, we used to throw it over to "Ops" and
| say, "your problem now."
|
| And Junior developers have always been overwhelmed with
| the details and under pressure to deliver enough to keep
| their job. None of this is new! I'm a graybeard now, but
| I remember seniors having the same complaints back then.
| "Kids these days" never gets old.
| bumby wrote:
| I get the pragmatism argument, but I would like to think
| certain professions should hold themselves to a higher
| standard. Doctors, lawyers, and engineers have a duty to
| society IMO that runs counter to a "just mail it in to
| cash a paycheck" mentality. I guess it comes down to
| whether you consider software developers to be that same
| kind of engineer. Certainly I don't want safety critical
| software engineers to have that cavalier attitude
| (although I've seen it).
| choilive wrote:
| Its been a while since my undergrad (>10 yrs) but many of
| my peers were majoring in CS or EE/CE because of the money,
| at the time I thought it was a bit depressing as well.
|
| With a few more years under my belt I realized theres
| nothing wrong with doing good work and providing
| yourself/your family a decent living. Not everyone needs
| the passion in their field to become among the best or
| become a "10x"er to contribute. We all have different
| passions, but we all need to pay the bills.
| neilv wrote:
| Yeah, I think that the _quality_ of work (skill +
| conscientiousness), and the _motivation_ for doing the
| work, are two separate things.
|
| Off-the-cuff, three groups:
|
| 1. There are people who are motivated by having a solid
| income, yet they take the professionalism seriously, and
| do skilled rock-solid work, 9-5. I'd be happy to work
| with these people.
|
| 2. There are other people who are motivated by having a
| solid or more-than-solid income, and (regardless of skill
| level), it's non-stop sprint performance art, gaming
| promotion metrics, resume-driven development, practicing
| Leetcode, and hopping at the next opportunity regardless
| of where that leaves the project and team.
|
| 3. Then there's those weirdos who are motivated by
| _something_ about the work itself, and would be doing it
| even if it didn 't pay well. Over the years, these people
| spend so much time and energy on the something, that they
| tend to develop more and stronger skills than the others.
| I'd be happy to work with these people, so long as they
| can also be professional (including rolling up sleeves
| for the non-fun parts), or amenable to learning to be
| professional.
|
| Half-joke: The potential of group #3 is threatening to
| sharp-elbowed group #2, so group #2 neutralizes them via
| frat gatekeeping tactics (yeah-but-what-school-did-you-
| go-to snobbery, Leetcode shibboleth for nothing but
| whether you rehearsed Leetcode rituals, cliques, culture
| fit, etc.).
|
| Startups might do well to have a mix of #3 and #1, and to
| stay far away from #2. But startups -- especially the
| last decade-plus of too many growth investment scams --
| are often run by affluent people who grew up being taught
| #2 skills (for how you game your way into prestigious
| school, aggressively self-interested networking and
| promoting yourself, etc.).
| dartos wrote:
| I'm in group 3 for sure.
|
| The hardest part of any job I've had is doing the not so
| fun parts (meetings, keeping up with emails, solidly
| finishing work before moving on to something new)
| neilv wrote:
| You're on the nominally right Web site. #1 is engineers,
| #2 is cancer, and #3 is hackers (possibly also
| engineers). :)
| sgarland wrote:
| I would definitely do this job even if it paid half as
| well. Nothing else is nearly as interesting to me, and
| I've tried quite a range.
| zquzra wrote:
| #3 is the old school hacker, "a high-powered mutant of
| some kind never even considered for mass production. Too
| weird to live, and too rare to die" :-)
| carlmr wrote:
| >Cloud was a mistake; it's inculcated people with the idea
| that chasing efficiency and optimization doesn't matter,
| because you can always scale up or out.
|
| Similarly Docker is an amazing technology, yet it enabled the
| dependency tower of babels that we have today. It enabled
| developers that don't care about cleaning up their depencies.
|
| Kubernetes is amazing technology, yet it enabled the
| developers that don't care to ship applications that
| constantly crash, but who cares, kubernetes will
| automatically restart everything.
|
| Cloud and now AI are similar enabler technologies. They could
| be used for good, but there are too many people that just
| don't care.
| shadowgovt wrote:
| The fine art of industry is building more and more
| elaborate complicated systems atop things someone deeply
| cares about to be used by those who don't have to deeply
| care.
|
| How many developers do we imagine even know the difference
| between SIMD and SISD operators, much less whether their
| software stack knows how to take advantage of SIMD? How
| many developers do we imagine even know how RAM chips store
| bits or how a semiconductor works?
|
| We're just watching the bar of "Don't need to care because
| a reliable system exists" move through something we know
| and care about in our lifetimes. Progress is great to watch
| in action.
| blibble wrote:
| > How many developers do we imagine even know the
| difference between SIMD and SISD operators, much less
| whether their software stack knows how to take advantage
| of SIMD? How many developers do we imagine even know how
| RAM chips store bits or how a semiconductor works?
|
| hopefully some of those that did a computer science
| degree?
|
| (all of what you've said was part of mine)
| shadowgovt wrote:
| Globally, only 41% of software developers even have a
| bachelor's degree.
| mistrial9 wrote:
| you insult fine art
| jcgrillo wrote:
| In a similar vein, some days I feel like a human link
| generator into e.g. postgres or kafka documentation. When
| docs are that clear, refined, and just _damn good_ but it
| seems like nobody is willing to actually read them closely
| enough to "get it" it's just a very depressing and
| demotivating experience. If I could never again have to
| explain what a transaction isolation level is or why calling
| kafka a "queue" makes no sense at all I'd probably live an
| extra decade.
| nerdjon wrote:
| I am curious about this also. I have now seen multiple PR's
| that I had to review that a method was clearly completely
| modified by AI with no good reason and when asked why something
| was changed we just got silence. Not exaggerating, literal
| silence and then trying to ignore the question and explain the
| thing we were first asking them to do. Clearly having no idea
| what is actually in this PR.
|
| This was done because we asked for a minor change to be done
| (talking maybe 5 lines of code) and tested. So now not only are
| we dealing with new debt, we are dealing with code that no one
| can explain why it was completely changed (and some of the
| changes were changes for the sake of change), and we are
| dealing with those of us that manage this code now looking at
| completely foreign code.
|
| I keep seeing this with people that are using these tools and
| they are not higher level engineers. We finally got to the
| point of denying these PR's and saying to go back and do it
| again. Loosing any of the time that was theoretically gained
| from doing it in the first place.
|
| Not saying these tools don't have a place. But people are using
| it without understanding what it is putting out and not
| understanding the long term effects it will have on a code
| base.
| okwhateverdude wrote:
| > Not saying these tools don't have a place. But people are
| using it without understanding what it is putting out and not
| understanding the long term effects it will have on a code
| base.
|
| It is worse than that. We're all maintaining in our heads the
| mental sand castle that is the system the code base
| represents. The abuse of the autocoder erodes that sand
| castle because the intentions of the changes, which are
| crucial for mentally updating the sand castle, are not
| communicated (because they are unknowable). This is same
| thing with poor commit messages, or poor documentation around
| requirements/business processes. With enough erosion, plus
| expected turn over in staff, the sand castle is actually
| gone.
| rongenre wrote:
| There needs to be a pattern for including AI-generated
| code, recording all the context which led to the
| generation.
| jcgrillo wrote:
| Easy: exclude developers who try it. Learning--be it a
| new codebase, a new programming language, a new database
| --takes time. You're not going to be as productive until
| you learn how. That's fine! Cheating on your homework
| with an LLM should not be something we celebrate, though,
| because the learner will never become productive that
| way, and they won't understand the code they're
| submitting for review.
| anonzzzies wrote:
| > Also, I've personally seen more interest in AI in devs that
| have little interest in technology, but a big interest in
| delivering.
|
| I found this too. But I also found the opposite, including here
| on HN; people who are interested in technology have almost an
| aversion against using AI. I personally love tech and I would
| and do write software for fun, but even that is objectively
| more fun for me with AI. It makes me far more productive (very
| much more than what the article states) and, more importantly,
| it removes the procrastination; whenever I am stuck or
| procrastinating getting to work or start, I start talking with
| Aider and before I know it, another task was done that I
| probably wouldn't have done that day without.
|
| That way I now launch bi weekly open and closed source projects
| while before that would take months to years. And the cost of
| having this team of fast experienced devs sitting with me is
| max a few $ per day.
| andybak wrote:
| Yeah. There is a psychological benefit to using AI that I
| find very beneficial. A lot of tasks that I would have
| avoided or wasted time doing task avoidance on suddenly
| become tractable. I think Simon Willison said something
| similar.
| jcgrillo wrote:
| Are you just working on personal projects or in a shared
| codebase with tens or hundreds of other devs? If the
| latter, how do you keep your AI generated content from
| turning the whole thing into an incomprehensible superfund
| site of tech debt? I've gotten a lot of mileage in my
| career so far by paying attention when something felt
| tedious or mundane, because that's a signal you need some
| combination of refactoring, tooling, or automation. If
| instead you just lean on an LLM to brute-force your way
| through, sure, that accomplishes the short term goal of
| shipping your agile sprint deliverable or whatever, but
| what of the long term cost?
| trashtester wrote:
| > people who are interested in technology have almost an
| aversion against using AI
|
| I wonder if this is an age thing, for many people. I'm old
| enough to have started reading these discussions on Slashdot
| in the early 90s.
|
| But between 2000 and 2010, Slashdot changed and became much
| less open to new ideas.
|
| The same may be happening to HN right now.
|
| It feels like a lot of people are stuck in the tech of 10
| years ago.
| infecto wrote:
| I hear this but I don't think this is a new issue that AI
| brought, it simply magnified it. That's a company culture
| issue.
|
| It reminds me of a talk Raymond Hettinger put on a while ago
| about rearranging the flowers in the garden. There is a
| tendency from new developers to rearrange for no good reason,
| AI makes it even easier now. This comes down to a culture
| problem to me, AI is simply the tool but the driver is the
| human (at least for now).
| noobermin wrote:
| It doesn't, it looks at PRs, commits, and builds only. They
| remark at this lacking context.
| fer wrote:
| Disclaimer: I work at a company who sells coding AI (among many
| other things).
|
| We use it internally and the technical debt is an enormous
| threat that IMO hasn't been properly gauged.
|
| It's very very useful to carpet bomb code with APIs and
| patterns you're not familiar with, but it also leads to insane
| amounts of code duplication and unwieldy boilerplate if you're
| not careful, because:
|
| 1. One of the two big bias of the models is the fact that the
| training data is StackOverflow-type training data, which are
| examples and don't take context and constraints into account.
|
| 2. The other is the existing codebase, and it tends to
| copy/repeat things instead of suggesting you to refactor.
|
| The first is mitigated by, well, doing your job and
| reviewing/editing what the LLM spat out.
|
| The second can only be mitigated once diffs/commit history
| become part of the training data, and that's a much harder
| dataset to handle and tag, as some changes are good
| (refactorings) but other might be not (bugs that get corrected
| in subsequent commits) and no clear distinction as commit
| messages are effectively lies (nobody ever writes: bug
| introduced).
|
| Not only that, merges/rebases/squashes alter/remove/add
| spurious meanings to the history, making everything blurrier.
| trashtester wrote:
| If you're in a company that is valued at $10M+ per developer,
| the technical debt is not a big concern.
|
| Either you will go bust, OR you will be able to hire enough
| people to pay those debts, once you get traction in the
| market.
| kraftman wrote:
| > Also, I've personally seen more interest in AI in devs that
| have little interest in technology, but a big interest in
| delivering. PMs love them though.
|
| Is this a bad thing? Maybe I'm misunderstanding it, but even
| when I'm working on my own projects, I'm usually trying to
| solve a problem, and the technology is a means to an end to
| solving that problem (delivering). I care that it works, and is
| maintainable, I don't care that much about the technology.
| never_inline wrote:
| Read it as "closing a ticket" that makes sense. They don't
| care if the code explodes after the sprint.
| layer8 wrote:
| Caring that it works and is maintainable is caring about the
| technology.
| add-sub-mul-div wrote:
| Maintaining bad AI code is the new maintaining bad offshore
| code.
| acedTrex wrote:
| > Also, I've personally seen more interest in AI in devs that
| have little interest in technology, but a big interest in
| delivering. PMs love them though.
|
| Bingo, this, so much this. Every dev i know who loves AI stuff
| was a dev that I had very little technical respect for pre AI.
| They got some stuff done but there was no craft or quality to
| it.
| chasd00 wrote:
| i'm certainly skeptical of genai but your argument sounds
| very familiar to assembler dev feelings towards c++ devs back
| in the day.
| acedTrex wrote:
| I am not necessarily arguing against GenAI. I am sure it
| will have somewhat similar effects to how the explosion in
| popularity of garbage collected languages et al had on
| software back in the 90s.
|
| More stuff will get done, the barrier of entry will be
| lower etc.
|
| The craft of programming took a significant quality/care
| hit when it transitioned from "only people who care enough
| to learn the ins and outs of memory management can feasibly
| do this" to "now anyone with a technical brain and a
| business use case can do it". Which makes sense, the code
| was no longer the point.
|
| The C++ devs rightly felt superior to the new java devs in
| the narrow niche of "ability to craft code." But that
| feeling doesn't move the needle business wise in the vast
| majority of circumstances. Which is always the schism
| between large technology leaps.
|
| Basically, the argument of "its worse" is not WRONG. Just,
| the same as it did not really matter in the mid 90s. Does
| not matter as much now, compared to the ability to "just
| get something that kinda works."
| wahnfrieden wrote:
| You're playing status games
| acedTrex wrote:
| I mean... in a way yes. The status of being someone who
| cares about the craft of programming.
|
| In the scheme of things however, that status hardly matters
| compared to the "ability to get something shipped quickly"
| which is what the vast majority of people are paid to do.
|
| So while I might judge those people for not meeting my
| personal standards or bar. In many cases that does not
| actually matter. They got something out there, thats all
| that matters.
| Delk wrote:
| For what it's worth, a (former) team mate who was one of the
| more enthusiastic adopters of gen AI at the time was in fact
| a pretty good developer who knew his stuff and wrote good
| code. He was _also_ big on delivering and productivity.
|
| In terms of directly generating technical content, I think he
| mostly used gen AI for more mechanical stuff such as drafting
| data schemas or class structures, or for converting this or
| that to JSON, and perhaps not so much for generating actual
| program code. Maybe there's a difference to someone who likes
| to have lots of program logic generated.
| acedTrex wrote:
| I have certainly used it for different mechanical things. I
| have copilot, pay for gpt4o etc.
|
| I do think there is a difference between a skilled engineer
| using it for the mechanical things, and an engineer that
| OFFLOADS thinking/designing to it.
|
| Theres nuance everywhere, but my original comment was
| definitely implying the people that attempt to lean on it
| very hard for their core work.
| trashtester wrote:
| Those bad engineers who offload critical thinking to
| copilot will be the first to leave....
| simonw wrote:
| I'm a developer who loves using AI, and I'm confident that I
| was a capable developer producing high quality code long
| before I started using AI tools.
| mistrial9 wrote:
| ok but you are not the sort that most of these comments are
| talking about.. Wasn't your demo at PyCon making jaws drop
| among hard core techies ?
| williamcotton wrote:
| Generalizing the talent and enthusiasm of your peers
| based on the tools they enjoy to use is going to cause
| friction.
| ben_w wrote:
| If it were not for all the things in your profile, I would
| aver that the only devs I know that think otherwise of
| their coding abilities were students at the time.
| Workaccount2 wrote:
| SWE has a huge draw because frankly it's not that hard to
| learn programming, and the bar to clear in order to land a
| $100-120k work-from-home salary is pretty low. I know more
| than a few people who career hopped into software engineering
| after a lackluster non-tech career (that they paid through
| the nose to get a degree in, but were still making $70k after
| 5 years). By and large these people seem to just not be "into
| it", and like you said are more about delivering than
| actually making good products/services.
|
| However, it does look like LLM's are racing to make these
| junior devs unnecessary.
| trashtester wrote:
| > However, it does look like LLM's are racing to make these
| junior devs unnecessary.
|
| The main utility of "junior devs" (regardless of age) is
| that they can serve as an interface to non-technical
| business "users". Give them the right tools, and their
| value will be similar to good business controllers or
| similar in the org.
|
| A salary of $100-$150k is really low for someone who is
| really a competent developer. It's kept down by those
| "junior devs" (of all ages) that apply for the same jobs.
|
| Both kinds of developers will be required until companies
| use AI in most of those roles, including the controllers,
| the developers and the business side.
| trashtester wrote:
| > was a dev that I had very little technical respect for pre
| AI
|
| There are two interpretations of this:
|
| 1) Those people are imposters
|
| 2) It's about you, not them
|
| I've been personally interested in AI since the early 80's,
| neural nets since the 90's, and vigilant about "AI" since
| Alexnet.
|
| I've also been in a tech lead role for the past ~25 years. If
| someone is talking about newer "AI" models in a nonsensical
| way, I cringe.
| jacobsenscott wrote:
| > Also, I've personally seen more interest in AI in devs that
| have little interest in technology, but a big interest in
| delivering. PMs love them though.
|
| You get what you measure. Nobody measure software quality.
| wahnfrieden wrote:
| Maybe not at your workplaces, but at mine, we measured bugs,
| change failure rate, uptime, "critical functionality" uptime,
| regressions, performance, CSAT, etc. in addition to
| qualitative research on quality in-team and with customers
| prophesi wrote:
| From my experience, most of it is quickly caught in code
| review. And after a while it occurs less and less, granted that
| the junior developer puts in the effort to learn why their PRs
| aren't getting approved.
|
| So, pretty similar to how it was before. Except that motivated
| junior developers will improve incredibly fast. But that's also
| kind of always been the case in software development these past
| two decades?
| YeGoblynQueenne wrote:
| It's interesting to see your comment currently right next to
| that of nerdjon here:
|
| https://news.ycombinator.com/item?id=41465827
|
| (the two comments might have moved apart by the time you read
| this).
|
| Edit: yep, they just did.
| Kiro wrote:
| I don't think it's that clear cut. I personally think the AI
| often delivers a better solution than the one I had in mind. It
| always contains a lot more safe guards against edge cases and
| other "boring" stuff that the AI has no problem adding but
| others find tedious.
| trashtester wrote:
| If you're building a code base where AI is delivering on the
| the details of it, it's generally a bad thing if the code
| provided by AI provide safeguards WITHIN your code base.
|
| Those kinds of safeguards should instead be part of the
| framework you're using. If you need to prevent SQL injection,
| you need to make sure that all access to the SQL type
| database pass through a layer that prevents that. If you are
| worried about the security of your point of access (like an
| API facing the public), you need to apply safeguards as close
| to the point of entry as possible, and so on.
|
| I'm a big believer in AI generated code (over a long
| horizon), but I'm not sure the edge case robustness is the
| main selling point.
| Kiro wrote:
| Sounds like we're talking about different kind of
| safeguards. I mean stuff like a character in a game ending
| up in a broken case due to something that is theoretically
| possible but very unlikely or where the consequence is not
| worth the effort. An AI has no problem taking those into
| account and write tedious safeguards, while I skip it.
| godelski wrote:
| > I wonder if the study includes the technical debt that more
| experienced developers had to tackle after the less experienced
| devs have contributed their AI-driven efforts.
|
| It does not
|
| You also may find this post from the other day more
| illuminating[0], as I believe the actual result strongly hints
| at what you're guessing. The study is high schoolers doing
| math. While GPT only has an 8% error rate for the final answer,
| it gets the steps wrong half the time. And with coding (like
| math), the steps are the important bits.
|
| But I think people evaluate very poorly when there's ill
| defined metrics but some metric exists. They over inflate it's
| value since it's concrete. Like completing a ticket doesn't
| mean you made progress. Introducing technical debt would mean
| taking a step back. A step forward in a very specific direction
| but away from the actual end goal. You're just outsourcing work
| to a future person and I think we like to pretend this doesn't
| exist because it's hard to measure.
|
| [0] https://news.ycombinator.com/item?id=41453300
| mrighele wrote:
| > Also, I've personally seen more interest in AI in devs that
| have little interest in technology, but a big interest in
| delivering. PMs love them though.
|
| For them programming is a means to an end, and I think it is
| fine, in a way. But you cannot just ask an AI to write you
| tiktok clone and expect to get the finished product. Writing
| software is an iterative process, and LLMs currently used are
| not good enough for that, because they need not only to answer
| the questions, but at the very minimum to start asking
| questions: "why do want to do that ?" "do you prefer this or
| that", etc., so that they can actually extract all the
| specification details that the user happily didn't even know he
| needed before producing an appropriate output. (It's not too
| different from how some independent developers has to handle
| their clients, isn't it ?). Probably we will get there, but not
| too soon.
|
| I also doubt that current tools can keep a project
| architecturally sound long-term, but that is just an hunch.
|
| I admit though that I may be biased because I don't like much
| tools like copilot: when I write software, I have in my mind a
| model of the software that I am writing/I want to write, the AI
| has another model "in mind" and I need to spend mental energy
| understanding what it is "thinking". Even if 99/100 it is what
| I wanted, the remaining 1% is enough to hold me back from
| trusting it. Maybe I am using it the wrong way, who knows.
|
| The AI tool that work for me would be a "voice controller AI
| powered pair programmer": I write my code, then from time to
| time I ask him questions on how to do something, and I can get
| either an contextual answer depending on the code I am working
| on, or generate the actual code if I wish so". Are there
| already plugins working that way for vscode/idea/etc ?
| liminalsunset wrote:
| I've been playing with Cursor (albeit with a very small toy
| codebase) and it does seem like it could do some of what you
| said - it has a number of features, not all of which
| necessarily generate code. You can ask questions about the
| code, about documentation, and other things, and it can
| optionally suggest code that you can either accept, accept
| parts of, or decline. It's more of a fork of vscode than a
| plugin right now though.
|
| It is very nice in that it gives you a handy diffing tool
| before you accept, and it very much feels like it puts me in
| control.
| mrighele wrote:
| Thanks, I might give it a try
| creativeSlumber wrote:
| This. Even without AI, we have inexperienced developers rolling
| out something that "just works" without thinking about many of
| the scaling/availability issues. Then you have to spend 10x the
| time fixing those issues.
| wouldbecouldbe wrote:
| I'm guiding a few and sometimes they write pretty good code with
| the help of GPT but then in meetings have trouble understanding
| and explaining things.
|
| I think it's a big productivity boost, but also a chance that the
| learning rate might actually be significantly slower.
| GoToRO wrote:
| For me AI just brought back documentation. All new frameworks
| lack documentation big time. The last good one for me was a DOS
| book! I don't think newer developers even have an idea of what
| good documentation looks like.
|
| Even so, AI will propose different things at different times and
| you still need an experienced developer to make the call. In the
| end it replaces documentation and typing.
| choilive wrote:
| I think AI just made documentation more valuable.
|
| For public facing projects - your documentation just became
| part of the LLM's training data, so its now extra important
| your documentation is thorough and accurate because you will
| have a ton of developers getting answers from that system.
|
| For private projects, your documentation can now be fed into a
| finetuning dataset or a RAG system, achieving the same effect.
| jakub_g wrote:
| Every now and then there's a HN discussion "how do you manage
| internal documentation" where most commenters write something
| like "there's no point writing documentation because it quickly
| becomes outdated". (There were two such threads in just last
| days). Might explain why nothing is documented anymore.
| tensor wrote:
| Yes, it can both help you write it but also, if you _start_
| with the documentation, e.g. the comment describing what
| function does, AI becomes orders of magnitude better and
| actually helping write the function, so it actually forces
| developers to better document their code!
| null_investor wrote:
| They also added lots of technical debt as I'm sure they used the
| AI to generate tests and some of those tests could be actually
| testing bugs as the correct behavior.
|
| I've already fixed a couple of tests like this, where people
| clearly used AI and didn't think about it, when in reality it was
| testing something wrong.
|
| Not to mention the rest of the technical debt added... looking at
| productivity in software development by amount of tasks is so
| wrong.
| onlyrealcuzzo wrote:
| > They also added lots of technical debt as I'm sure they used
| the AI to generate tests and some of those tests could be
| actually testing bugs as the correct behavior.
|
| Let's not forget that developers some times do this, too...
| randomdata wrote:
| _> when in reality it was testing something wrong._
|
| Must have seen AI write the implementation as well?
|
| If you're still cognizant of what you're writing on the
| implementation side, it's pretty hard to see a test go from
| failing to passing if the test is buggy. It requires you to
| independently introduce the same bug the LLM did, which, while
| not completely impossible, is unlikely.
|
| Of course, humans are prone to not understanding the
| requirements, and introducing what isn't really a bug in the
| strictest sense but rather a misfeature.
| Jensson wrote:
| > it's pretty hard to see a test go from failing to passing
|
| Its pretty easy to add a passing test and call it done
| without checking if it actually fails in the right
| circumstances, and then you will get a ton of buggy tests.
|
| Most developers don't do the start out at failing and then to
| passing ritual, especially junior ones who copies code from
| somewhere instead of knowing what they wrote.
| muglug wrote:
| This is a decently-thorough study, using PRs as a productivity
| metric while also tracking build failures (which remained
| constant at MSFT but increased at Accenture).
|
| Would love to see it replicated by researchers at a company that
| does not have a clear financial interest in the outcome (the
| corresponding author here was working at Microsoft Research
| during the study period).
|
| > Before moving on, we discuss an additional experiment run at
| Accenture that was abandoned due to a large layoff affecting 42%
| of participants
|
| Eek
| lolinder wrote:
| The most interesting thing about this study for me is that when
| they break it down by experience levels, developers who are above
| the median tenure show no statistically significant increase in
| 'productivity' (for some bad proxies of productivity), with the
| 95% confidence intervals actually dipping deep into the negatives
| on all metrics (though leaning slightly positive).
|
| This tracks with my own experience: Copilot is nice for resolving
| some tedium and freeing up my brain to focus more on deeper
| questions, but it's not as world-altering as junior devs describe
| it as. It's also frequently subtly wrong in ways that a newer dev
| wouldn't catch, which requires me to stop and tweak most things
| it generates in a way that a less experienced dev probably
| wouldn't know to. A few years into it I now have a pretty good
| sense for when to use Copilot and when not to--so I think it's
| probably a net positive for me now--but it certainly wasn't
| always that way.
|
| I also wonder if the possibly-decreased 'productivity' for more
| senior devs stems in part from the _increase_ in 'productivity'
| from the juniors in the company. If the junior devs are producing
| more PRs that have more mistakes and take longer to review, this
| would potentially slow down seniors, reducing their own
| productivity gains proportionally.
| sum1ren wrote:
| When I'm using genai to write some code for me, I lose the
| internal mental state of what my code is doing.
|
| As such, when I do have to debug problems myself, or dream up
| ideas of improvements, I no longer can do this properly due to
| lack of internal mental state.
|
| Wonder how people who have used genai coding successfully get
| around this?
| heisenberg1 wrote:
| This is obvious. Right, of course you get an increase in
| productivity, especially as a junior - when current AI is able to
| solve leetcode.
|
| BUT I think a lot of people mentioned that, you get code - that
| the person which wrote it do not understand. So the next time you
| get a bug there, good luck fixing it.
|
| My take so far. AI is great, but only for non critical, non core
| code. Everything that is done for plotting and scripting is
| awesome (which can take days to implement and in minutes with AI)
| - but core lib functions - wouldn't outsource it to the AI right
| now.
| noobermin wrote:
| Can someone potentially smarter than me explain how the data,
| which in table I clearly shows the majority of the means for each
| experiment metric being _less_ than the SD could even hope to be
| salvaged? Taken blindly, the results are simply unbelieveable to
| outright lying, the sort of thing you see submitted to garbage
| open access journals. The text describing model they employ
| afterwards is not convincing enough for me and seems light on
| details. I mean, wouldn 't any reasonable reviewer demand more?
|
| I know preprints don't need polish but this is even below the
| standard of a preprint, imo.
| smokel wrote:
| At least they acknowledge this:
|
| "However, the table also shows that for all outcomes (with the
| exception of the Build Success Rate), the standard deviation
| exceeds the pre-treatment mean, and sometimes by a lot. This
| high variability will limit our power in our experimental
| regressions below."
|
| What I find even stranger is that the values in the "control"
| and "treatment" columns are so similar. That would be highly
| unlikely given the extreme variability, no?
| smokel wrote:
| Interestingly, they compare number of pull requests as a
| statistic for productivity. Not that I know of a better metric,
| but I wonder if this is an accurate metric. It seems similarly
| misguided as looking at lines of code.
|
| If an AI tool makes me more productive, I would probably either
| spend the time won browsing the internet, or use it to attempt
| different approaches to solve the problem at hand. In the latter
| case, I would perhaps make more reliable or more flexible
| software. Which would also be almost impossible to measure in a
| scientific investigation.
|
| In my experience, the differences in developer productivity are
| so enormous (depending on existing domain knowledge, motivation,
| or management approach), that it seems pretty hard to make any
| scientific claim based on looking at large groups of developers.
| For now, I prefer the individual success story.
| ergonaught wrote:
| The result is "less experienced people got more stuff done". I do
| not see an assessment of whether the stuff that got done was well
| done.
|
| The output of these tools today is unsafe to use unless you
| possess the ability to assess its correctness. The less able you
| are to perform that assessment, the more likely you are to use
| these tools.
|
| Only one of many problems with this direction, but gravity sucks,
| doesn't it.
| arexxbifs wrote:
| My hunch - it's just a hunch - is that LLM-assisted coding is
| detrimental to one's growth as a developer. I'm fairly certain it
| can only boost productivity to a certain level - one which may be
| tedium for more senior developers, but formative for juniors.
|
| My experience is that the LLM isn't just used for "boilerplate"
| code, but rather called into action when a junior developer is
| faced with a fairly common task they've still not (fully)
| understood. The process of experimenting, learning and
| understanding is then largely replaced by the LLM, and the real
| skill becomes applying prompt tweaks until it looks like stuff
| works.
| arealaccount wrote:
| Why I recommend something like CoPilot over ChatGPT to juniors
| wanting to use AI.
|
| At least Copilot you're still more or less driving, whereas
| ChatGPT you're more the passenger and not growing intuition.
| DanHulton wrote:
| I was really hoping this study would be exploring that. Really,
| it's examining short-term productivity gains, ignoring long-
| term tech debt that can occur, and _completely_ ignoring
| effects on the growth of the software developers themselves.
|
| I share your hunch, though I would go so far as to call it an
| informed, strong opinion. I think we're going to pay the price
| in this industry in a few years, where the pipeline of "clueful
| junior software developers" is gonna dry way up, replaced by a
| firehose of "AI-reliant junior software developers", and the
| distance between those two categories is a GULF. (And of
| course, it has a knock-on effect on the number of clueful
| intermediate software developers, and clueful senior software
| developers, etc...)
| diob wrote:
| I think it can be abused like anything else (copy paste from
| stack overflow until it works).
|
| But for folks who are going to be successful with or without
| it, it's a godsend in terms of being able to essentially ask
| stack overflow questions and get immediate non judgemental
| answers.
|
| Maybe not correct all the time, but that was true with stack
| overflow as well. So as always, it comes back to the
| individual.
| throwaway314155 wrote:
| My experience has been that indeed, it is detrimental to
| juniors. But unlike your take, it is largely a boon to
| experienced developers. That you suggest "tedium" is involved
| for more senior developers suggests to me that you haven't
| given the tooling a fair chance or work with a relatively
| obscure technology/language.
| layer8 wrote:
| I think you've misunderstood the GP. They are saying AI is
| useful to seniors for tasks that would otherwise be tedious,
| but doing those tedious tasks by hand would be formative for
| juniors, and it is detrimental to their growth when they do
| them using AI.
| Workaccount2 wrote:
| With the progress LLM's having been making in the last two
| years, is it actually a bad bet to not want to really get into
| it?
|
| How many contemporary developers have no idea how to write
| machine code, when 50 years ago it was basically mandatory if
| you wanted to be able to write anything?
|
| Are LLM's just going to become another abstraction crutch
| turned abstraction solid pillar?
| arexxbifs wrote:
| Abstraction is beneficial and profitable up to a certain
| point, after which upkeep gets too hard or expensive, and
| knowledge dwindles into a competency crisis - for various
| reasons. I'm not saying we are at that point yet, but it
| feels like we're closing in on it (and not just in software
| development). 50 years ago isn't even 50 years ago anymore,
| if you catch my drift: In 1974, the real king of the hill was
| COBOL - a very straight-forward abstraction.
|
| I'm seeing a lot of confusion and frustration from beginner
| programmers when it comes to abstraction, because a lot of
| abstractions in use today just incur other kinds of
| complexity. At a glance, React for example can seem
| deceptively easy, but in truth it requires understanding of a
| lot of advanced concepts. And sure, a little knowledge can go
| a long way in E.G. web development, but to really write
| robust, performant code you have to know a lot about the
| browser it runs in, not unlike how great programmers of
| yesteryear had entire 8-bit machines mapped out in their
| heads.
|
| Considering this, I'm not convinced the LLM crutch will ever
| solidify into a pillar of understanding and maintainable
| competence.
| jprete wrote:
| LLMs aren't an abstraction, even a very leaky one, so the
| analogies with compilers and the like really fall flat for
| me.
| tensor wrote:
| I think it really depends on the users. The same people who
| would just paste stackoverflow code until it seems to work and
| call it a day will abuse LLMs. However, those of us who like to
| know everything about the code we write will likely research
| anything an LLM spits out that we don't know about.
|
| Well, at least that's how I use them. And to throw a counter to
| your hypothesis, I find that sometimes the LLM will use
| functions or library components that I didn't know of, which
| actually saves me a lot of time when learning a new language or
| toolkit. So for me, it actually accelerates learning rather
| than retarding it.
| pphysch wrote:
| It all depends of one is using it scientifically, i.e. having a
| hypothesis for the generated code before it exists, so that you
| can evaluate it.
|
| When used scientifically, coding copilots boost productivity
| AND skills.
| 29athrowaway wrote:
| The study doesn't cover the long term effects of using generative
| AI on a project, which is the deskilling of your workforce.
|
| Because development will become an auction-like activity where
| the one that accepts more suggestions wins.
| wrink wrote:
| I think this is right. Just have to jump ship before it all
| explodes and people wonder what happened
|
| Similar to gaming the stock price for a couple quarters as
| C-level, but now with more incentive for this sort of behavior
| at IC level
| einpoklum wrote:
| Reminds me of a situation I've been in a few times already:
| Dev: Hey einpoklum, how do I do XYZ? Me: Hmm, I think I
| remember that... you could try AB and then C. Dev: Ok, but
| isn't there a better/easier way? Let me ask ChatGPT. ...
| Dev: Hey einpoklum, ChatGPT said I should do AB and then C.
| Me: Let me have a look at that for a second. Me: -Right,
| so it's just what I read on StackOverflow about this, a couple of
| years ago.
|
| Sometimes it's even the answer that _I_ wrote on StackOverflow
| and then I feel cheated.
| slt2021 wrote:
| true, ChatGPT even imports libraries and configs that are 4-5
| year outdated.
| einpoklum wrote:
| I didn't mean to say the answer is outdated... I try not to
| give people outdated answers :-\
| zitterbewegung wrote:
| I am an average developer with more than five years experience in
| Python. I was using chatgpt to create prototypes of what to do in
| something I was familiar with and was able to debug the task to
| make it work. I wasn't specific enough to specify the epaper
| display had seven colors instead of black and white.
|
| When I was using chatgpt to do qualifiers for a CTF called Hack A
| Sat at defcon 31 I could not get anything to work such as gnu
| radio programs.
|
| If you have the ability to debug then I have experienced that it
| is productive but when you don't understand you run into
| problems.
| yieldcrv wrote:
| > Notably, less experienced developers showed higher adoption
| rates and greater productivity gains.
|
| This is what I've seen too, I don't think less experienced
| developers have gotten better in their understanding of anything
| just more exposed and quicker, while I do think more experience
| developers have stagnated
| dpflan wrote:
| > while I do think more experience developers have stagnated
|
| Is this because they are not using coding assistants? Are they
| resistant to using them? I have to say that the coding
| assistant is helpful; it is an ever-present rubber duck that
| can talk back with useful information.
| yieldcrv wrote:
| Across teams and organizations what I've seen this year is
| that the "best" developers haven't looked. The proverbial
| rockstars with the most domain knowledge have never gotten
| around to AI and LLMs at all.
|
| This is compounded by adherence to misguided corporate
| policies that broadly prohibit use of LLMs but are meant to
| only be about putting trade secrets into the cloud, not
| distinguishing between cloud vs locally run language models.
| Comfortable people would never challenge this policy with
| critical thinking, and it requires special interest to look
| at locally run language models, even just to choose which one
| to run.
|
| Many developers have not advocated for more RAM on their
| company issued laptop to run better LLMs.
|
| and I haven't seen any internal language model that the
| company is running in their intranet. But it would be cool if
| there was a huggingface-style catalogue and server farm
| companies could have and let their employees choose models to
| prompt, always having the latest models to load.
| simpleranchero wrote:
| Microsoft people in research team proving Microsoft tools are
| good. Elsevier will now do ads in research papers.
| Bjorkbat wrote:
| This is far, far more rigorous than the experiment Microsoft
| behind Microsoft's claim that Copilot made devs 55% faster.
|
| The experiment in question was to split 95 devs into two groups
| and see how long it took each group to setup a web server in
| Javascript. Control took a little under 3 hours on average, the
| copilot group took 1 hour and 11 minutes on average.
|
| https://github.blog/news-insights/research/research-quantify...
|
| And it is thanks to this weak experiment that Github proudly
| boasts that Copilot makes devs 55% faster.
|
| By contrast the conclusion that Copilot makes devs ~25% more
| productive seems reasonable, especially when you read the
| actual paper and find out that among senior devs the
| productivity gains are more marginal.
| mistrial9 wrote:
| what do these three things have in common?
|
| * control the agenda items in a formal meeting
|
| * fill a fixed amount of time in an interview with no
| rebuttal
|
| * design the benchmark experiments and the presentation of
| the results
| ang_cire wrote:
| I sometimes wonder about whether the decline in IT worker quality
| is down to companies trying to force more and more roles onto
| each worker to reduce headcount.
|
| Developers, Operations, and Security used to be dedicated roles.
|
| Then we made DevOps and some businesses took that to mean they
| only needed 2/3 of the headcount, rather than integrating those
| teams.
|
| Then we made DevSecOps, and some businesses took that to mean
| they only needed 1/3 the original roles, and that devs could just
| also be their operations and appsec team.
|
| That's not a knock on shift-left and integrated operations
| models; those are often good ideas. It's just the logical outcome
| of those models when execs think they can get a bigger bonus by
| cutting costs by cutting headcounts.
|
| Now you have new devs coming into insanely complex n-microservice
| environments, being asked to learn the existing codebase, being
| asked to learn their 5-tool CI/CD pipelines (and that ain't being
| taught in school), being asked to learn to be DBAs, and also to
| keep up a steady code release cycle.
|
| Is anyone really surprised they are using ChatGPT to keep up?
|
| This is going to keep happening until IT companies stop cutting
| headcounts to make line go up (instead of good business
| strategy).
| yifanl wrote:
| They're cutting headcount because they have no conception of
| how to make a good product, so reducing costs is the only of
| making the bottom line go up.
|
| That's across the board, from the startups whose business plan
| is to be acquired at all costs, to the giant tech companies,
| whose business plan is to get monopoly power first, then figure
| out how to extract money later.
| goatlover wrote:
| Failures of capitalism that probably need regulation. If the
| middle class and workers are to be protected.
| 01HNNWZ0MV43FF wrote:
| Would UBI and LVT do the trick? I'm okay with minimum wages
| on the idea that a democratic government is the biggest
| possible de facto labor union, but in general I don't want
| the government making too many rules about how you can hire
| and fire people
| WalterBright wrote:
| The city of Seattle decided that gig delivery drivers
| needed to be protected. They passed a minimum wage for
| them. The income of gig drivers promptly dropped, as the
| rise in prices corresponded with a drop in customers.
|
| Who regulates the regulators?
| malfist wrote:
| You need a citation for that claim. And something more
| than just "the week after this went into effect it
| reduced sale volume"
|
| And besides, what's the alternative here? Don't protect
| employees? Let them work below minimum wage?
| djbusby wrote:
| Why did the prices for Uber/Lyft rise everywhere else
| too?
|
| It wasn't Seattle laws that did that.
|
| It's that those companies needed to go from Growth-Mode
| to Margin-Mode. They could no longer sell VC dollars for
| dimes.
| numpad0 wrote:
| Yeah this after all. Private corporations exist to make
| money. They minimize investment and maximize returns. They
| progressively suppress wages and inflate profits. Clever
| individual choices and self discounts merely create
| temporary competitive advantages against other poor
| laborers in the market.
|
| If you manage to secure the bag that way and exit that
| class altogether, good for you, but that solves nothing for
| the rest. There's no be all end all threshold that everyone
| can just stay above and stay ahead of inflation.
| WalterBright wrote:
| > They're cutting headcount because they have no conception
| of how to make a good product, so reducing costs is the only
| of making the bottom line go up.
|
| The field is wide open for a startup to do it right. Why not
| start one?
| malfist wrote:
| Because the field isn't even. Monopoly or oligopoly power
| works to prevent new competitors while at the same time
| stagnating their offerings.
|
| Look at the explosion of browsers and their capabilities
| after the IE monopoly was broken.
| Buttons840 wrote:
| Well, either (1) our free market is not healthy and the
| ability to form new companies that do better is weak and
| feeble, or (2) the current companies are doing about as
| well as possibe and it is hard to out compete them.
|
| If it's the latter (number 2), then we need to start asking
| why American companies "doing about as well as possible"
| are incapable of producing secure and reliable software.
| Being able to produce secure and reliable software, in
| general and en masse, seems like a useful ability for a
| nation.
|
| Reminds me of our national ability to produce aircraft;
| how's the competition in that market working out? And are
| we getting better or worse at producing aircraft?
| jajko wrote:
| So its not going to stop. Typical C-suite who holds real power
| has absolutely 0 clue about IT complexity, we are overpriced
| janitors to them. Their fault, their blame, but they are
| probably long gone when these mistakes manifest fully.
|
| In my banking corp, in past 13 years I've seen massive rise of
| complexity, coupled with absolutely madly done bureaucracy
| increase. I still could do all stuff that is required but - I
| dont have access. I cant have access. Simple task became 10
| steps negotiating with obscure Pune team that I need to chase
| 10x and escalate till they actually recognize there is some
| work for them. Processes became beyond ridiculous, you start
| something and it could take 2 days or 3 months, who knows.
| Every single app will break pretty quickly if not constantly
| maintained - be it some new network stuff, unchecked unix
| update, or any other of trillion things that can and will go
| wrong.
|
| This means - paper pushers and folks at best average at their
| primary job (still IT or related) got very entretched in
| processes and won, and business gets served subpar IT, projects
| over time and thus budget, perpetuating the image of shitty
| tolerated evil IT.
|
| I stopped caring, work to live is more than enough for me, that
| 'live' part is where my focus is and life achievements are.
| godelski wrote:
| - my laundry app (that I must use) takes minutes to load,
| doesn't cache my last used laundry room, and the list of
| rooms isn't even fucking sorted (literally: room 3, room 7,
| room 1, room 2, ...)
|
| - my AC's app takes 45 s to load even if I just used it,
| because it needs to connect. Worse, I'll bring the temp down
| in my house and in the evening raise it, but it'll come on
| even when 5F below my target value, staying on for 15+
| minutes leaving us freezing (5F according to __it's
| thermometer__!)
|
| - my TV controls are slow. Enough that I buffer inputs and
| wait 2-3 seconds for the commands to play. That pressing the
| exit button in the same menu (I turn down brightness at night
| because auto settings don't work, so it's the exact same
| behavior), idk if I'm exciting to my input, exiting the menu,
| or just exiting the sub menu. It's inconsistent!
|
| There's so much that I can go on and on and I'm sure you can
| too. I think one of the worst parts about being a programmer
| is that I'm pretty sure I know how to solve many of these
| issues, and in fact sometimes I'll spend days to tear apart
| the system to actually fix it. Of course to only be undone by
| updates that are forced (app or whatever won't connect
| because why is everything done server side ( + * D * ) + Shan
| + - + ). Even worse, I'll make PRs on open projects (or open
| issues another way and submit solutions) that having been
| working for months and they just go stale while I see other
| users reporting the same problems and devs working on other
| things in the same project (I'll even see them deny the
| behavior or just respond "works for me" _closes issue before
| opener can respond_ )
|
| I don't know how to stop caring because these things directly
| affect me and are slowing me down. I mean how fucking hard is
| it to use sort? It's not even one line!
|
| What the fuck is wrong with us?
| mistrial9 wrote:
| ... it is not "us" .. there is a collection of roles that
| each interact to produce products to market or releases.
| Add some game-theory to the thinking on that?
| godelski wrote:
| "Us" can mean different things depending on the context.
| The context here is "programmers" not "shitty
| programmers" (really we can be inclusive with engineers
| and any makers). Considering that it seems you appear to
| understand the context I'm a bit confused at the comment.
| You don't have to inject yourself if you're not in the
| referred to group (i.e. maybe I'm not talking about you.
| I can't know because I don't even know you)
|
| I need to stress so much -- especially because how people
| think about AI and intelligence (my field of research[0])
| -- that language is not precise. Language is not
| thoughts. Sure, many think "out loud" and most people
| have inner speech, but language (even inner) is
| compressed. You use language to convey much more complex
| and abstract concepts in your head to someone who is
| hopefully trying to make their decompression adapt to
| uncover the other person's intended meaning. This isn't a
| Chinese room where you just look in the dictionary (which
| also shows multiple definitions for anything). I know
| it's harder with text and disconnected cultures
| integration, but we must not confuse this and I think
| it's ignored far too often. Ignoring it seems to just
| escalate that problems. > Add some game-
| theory to the thinking on that?
|
| And please don't be dismissive by hand waving. I'm sure
| you understand game theory. Which would mean you
| understand what a complex problem this is to actually
| model (to a degree I doubt anyone could to a good degree
| of accuracy). That you understand perturbation theory and
| chaos theory and how they are critical to modeling such a
| process meaning you only get probability distribution as
| results.
|
| [0] I add this context because it decreases the amount of
| people that feel the need to nerdsplain things to me that
| they don't research.
| linotype wrote:
| I use an Apple TV. It's more expensive than Google and
| Amazon's offerings, but fast as hell and gets better with
| time.
| godelski wrote:
| I mean this is my TV menu. I'm adjusting brightness (the
| explicit example given). Does Apple TV control the whole
| TV settings?
|
| Fwiw, my TV just ends up being a big monitor because all
| the web apps are better and even with all the issues
| jellyfin has, it's better than many of those. I just
| mostly use a mouse or kdeconnect.
|
| Speaking of which, does anyone have a recommendation for
| an android keyboard that gives me things like Ctrl and
| super keys? Also is there a good xdotool replacement for
| Wayland? I didn't find ydotool working as well but maybe
| I should give it another try.
|
| I can suggest this setup and think it'll work for many.
| My desktop sits behind my TV because it mostly does
| computational work, might run servers, or gaming. I'm a
| casual and so 60fps 4k is more than enough even with the
| delay. Then I just ssh from my laptop and do most of the
| work from there. Pretty much the same as my professional
| work, since I need to ssh into hpc clusters, there's
| little I need to do on the machine I'm physically in
| front of (did we go back to the 70's?)
| smokel wrote:
| _> What the fuck is wrong with us?_
|
| This is simple: we can't just trust each other. When
| programming started, people were mostly interested in
| building things, and there was little incentive to spoil
| other peoples work. Now there is money to be made, either
| through advertising, or through malpractice. This means
| that people have to protect their code from others. Program
| code is obfuscated (compiled and copyright enforced) or
| stored in a container with a limited interface (cloud).
|
| It's not a technical issue, it's a social issue.
|
| Applying a social mindset to technical issues (asking your
| compiler to be your partner, and preparing them a nice
| dinner) is equally silly as applying a technical mindset to
| social issues.
| godelski wrote:
| > When programming started, people were mostly
| interested in building things, and there was little
| incentive to spoil other peoples work. Now there is money
| to be made, either through advertising, or through
| malpractice
|
| Yeah, I lean towards this too. Signals I use now to
| determine good software usually are things that look
| auxiliary because I'm actually looking for things that
| tell me the dev is passionate and "having fun." Like
| easter eggs, little things like it looks like they took
| way too much time to make something unimportant pretty
| (keeping doing this devs. I love it and it's appreciated.
| Always makes me smile ^_^). But I am also sympathetic,
| because yeah I also get tons of issues opened that should
| have been a google search or are wildly inappropriate.
| Though I try to determine if these are in good faith
| because we don't get wizards without noobs, and someone's
| got to teach them.
|
| But it all makes me think we forgot what all of this is
| about, even "the economy." Money is supposed to be a
| proxy for increasing quality of life. Not even just on a
| personal level. I'm happy that people can get rich doing
| their work and things that they're passionate about but I
| feel that the way current infrastructures are we're
| actively discouraging or handcuffing people who are
| passionate. Or that we try to kill that passion.
| Managers, let your devs "have fun." Reign them in so that
| they don't go too deep of rabbit holes and pull too far
| away, but coding (like any engineering or any science) is
| (also) an art. When that passion dies, enshitification
| ensues.
|
| For a concrete example: I'm wildly impressed that 99.9
| times I'm filling out a forum that includes things like a
| country or timezone that my country isn't either
| autoselected or a copy isn't located at the top (not
| moved! copied!). It really makes me think that better
| than chasing leet code questions for interviews you ask
| someone to build a simple thing and what you actually
| look for is the details and little things that make the
| experience smoother (or product better). Because it is
| hard to teach people to about subtly, much harder than
| teaching them a stack or specific language (and if they
| care about the little things they'll almost always be
| quicker to learn those things). Sure, this might take a
| little more work to interview people and doesn't have a
| precise answer, but programs don't have precise answers
| either. And given how long and taxing software interviews
| are these days I wouldn't be surprised if slowing down
| actually ends up speeding things up and saving a lot of
| money.
| pojzon wrote:
| Those are all small symptoms of late stage capitalism and
| closing on end of an era.
|
| Like Rome, corruption, laziness and barbarians will tear it
| all down.
| godelski wrote:
| I don't need help identifying the problem, I need help
| solving the problem.
| GeoAtreides wrote:
| > 10 steps negotiating with obscure Pune team that I need to
| chase 10x
|
| why are you doing the chasing? unless you're the project
| manager, comment "blocked by obscure pune team" on the ticket
| and move on
| malfist wrote:
| In my experience, PMs rarely do the chasing down for you.
| Most of them ask why you're still blocked and if you've
| done anything about it. Even if they do do something to
| chase them down, you're still on the hook for the
| deadlines.
| lispisok wrote:
| Like most things the decline in quality is probably multi-
| faceted. There is also the component where tech became a hot
| attractive field so people flooded in who only cared about the
| paycheck and not the craft.
| trashtester wrote:
| That definitely happend in the dotcom bubble. Plenty of
| "developers" were crowding the field, many of which neither
| had any real technical ability or interest.
|
| The nerds who were into programming based on personal
| interest were really not affected.
|
| Those who have tech as a passion will generally outpeform
| those who have it as a job, by a large margin.
|
| But salary structures tend to ignore this.
| stuckkeys wrote:
| Sadly. It is an onward trend. I have become so discouraged from
| this subject, that I am evaluating my career choices. Farming
| is a the safest bet for now.
| stuckinhell wrote:
| i'm worried this is the case as well
| huuhee3 wrote:
| Nursing is pretty safe too, simply due to demographics and
| limitations of robotics.
| malfist wrote:
| Have you actually done farming? I grew up on a farm. It's no
| where close to a safe bet.
|
| It's capital intensive, high risk, hard work with low
| margins. Not at all like stardew.
| djbusby wrote:
| Personal horticulture is fun and rewarding! Grow your own
| lettuce, tomato, etc.
|
| BigAg is HARD WORK
| trashtester wrote:
| They're realizing that 10x (+) developers exist, but think they
| can hire them at 1x developer salaries.
|
| Btw, they key skill you're leaving out, is to understand the
| business your company is in.
|
| If you can couple even moderate developer ability with a good
| understanding of business objectives, you may stay relevant
| even while some of the pure developers are replaced by AI.
| swader999 wrote:
| This 100%. It's rare to find anyone that wants to learn a
| complex biz domain.
| extr wrote:
| It's because it's often a suboptimal career move from an
| individual perspective. By branding yourself as a "FinTech
| Developer" or whatever instead of just a "Developer" you're
| narrowing your possible job options for questionable
| benefit in terms of TC and skills growth. This isn't always
| the case and of course if you spend 20 years in one domain
| maybe you can brand yourself as an expert there and pull
| high $/hr consulting rates for fixing people's problems.
| Maybe. More likely IMO you end up stagnating.
|
| I went through this myself early in my career. I did ML at
| insurance companies and got branded as an insurance ML guy.
| Insurance companies don't pay that well and there are a
| limited number of them. After I got out of that lane and
| got some name-brand tech experience under my belt, job
| hunting was much easier and my options opened up. I make a
| lot more money. And I can always go back if I really want
| to.
| trashtester wrote:
| > I did ML at insurance companies and got branded as an
| insurance ML guy.
|
| If you're an "ML insurance guy" outside of the US, it may
| be quite lucrative compared to other developers. It's
| really only in the US (and maybe China) that pure
| developers are demanding $200k+ salaries.
|
| In most places in Europe, even $100k is considered a high
| salary, and if you can provide value directly to the
| business, it will add a lot to your potential
| compensation.
|
| And in particular, if your skills are more about the
| business than the tech/math of your domain, you probably
| want to leverage that, rather than trying to compete with
| 25-year-olds with a strong STEM background, but poor
| real-life experience.
| matrix2003 wrote:
| > In most places in Europe, even $100k is considered a
| high salary
|
| I think it's worth adding here that US developers can
| have a much higher burden for things like health
| insurance. My child broke his arm this year, and we hit
| our very high deductible.
|
| I would like to see numbers factoring in things like
| public transportation, health insurance, etc., because I
| personally feel like EU vs US developers are a lot closer
| in quality of life after all the deductions.
| ghaff wrote:
| I'm skeptical.
|
| I live in a location that wouldn't have public
| transportation even in Europe. And my healthcare, while
| not "free," was never expensive outside of fairly small
| co-pays and a monthly deduction that wasn't cheap but was
| in the hundreds per month range. Of course, there are
| high deductible policies but that's a financial choice.
| WalterBright wrote:
| > got branded as an insurance ML guy
|
| I got branded as a compiler guy. My salary is $0.
| godelski wrote:
| B(i)ased as fuck.
|
| I think this post from the other day adds some important
| context[0]. In that study kids with access to GPT did way more
| practice problems but worse on the test. But the most important
| part was that they found that while GPT usually got the final
| answer right that the logic was wrong, meaning that the answer is
| wrong. This is true for math and code.
|
| There's the joke: there's two types of 10x devs, those that do
| 10x work and those who finish 10x jira tickets. The problem with
| this study is the assumptions that it makes, which is quite
| common and naive in our industry. They assume that PRs and
| commits are measures of productivity and they assume passing
| review is a good quality metric. These are so variable between
| teams. Plenty are just "lgtm" reviews.
|
| The issue here is that there's no real solid metric for things
| like good code. Meeting the goals of a ticket doesn't mean you
| haven't solved the problem so poorly you are the reason 10 new
| tickets will be created. This is the real issue here and the only
| real way to measure it is using Justice Potter's test (I know it
| when I see it), and requires an expert evaluator. In other words,
| tech debt. Which is something we're seeing a growing rise in, all
| the fucking enshitification.
|
| So I don't think that study here contradicts [0], in fact I think
| they're aligned. But I suspect people who are poor programmers
| (or non programmers) will use this at evidence for what they want
| to see. Believing naive things like lines of code, number of
| commits/PRs, etc are measures of productivity rather than hints
| of measure. I'm all for "move fast and break things" as long as
| there's time set aside to clean up the fucking mess you left
| behind. But there never is. It's like we have businesses ADHD.
| There's so much lost productivity because so much focus is placed
| on short term measurements and thinking. I know medium and long
| term thinking are hard, but humans do hard shit every day. We can
| do a lot better than a shoddy study like this.
|
| [0] https://news.ycombinator.com/item?id=41453300
| Delk wrote:
| It's probably worth going a bit deeper into the paper before
| picking up conclusions. And I think the study could really do a
| bit of a better job of summarizing its results.
|
| The abstract and the conclusion only give a single percentage
| figure (26.08% increase in productivity, which probably has too
| many decimals) as the result. If you go a bit further, they give
| figures of 27 to 39 percent for juniors and 8 to 13 percent for
| seniors.
|
| But if you go deeper, it looks like there's a lot of variation
| not only by seniority, but also by the company. Beside pull
| requests, results on the other outcome measures (commits, builds,
| build success rate) don't seem to be statistically significant at
| Microsoft, from what I can tell. And the PR increases only seem
| to be statistically significant for Microsoft, not for Accenture.
| And even then possibly only for juniors, but I'm not sure I can
| quite figure out if I've understood that correctly.
|
| Of course the abstract and the conclusion have to summarize. But
| it really looks like the outcomes vary so much depending on the
| variables that I'm not sure it makes sense to give a single
| overall number even as a summary. Especially since statistical
| significance seems a bit hit-and-miss.
|
| edit: better readability
| baxtr wrote:
| Re 26.08%: I immediately question any study (outside of physics
| etc) that provides two decimal places.
| kkyr wrote:
| Why?
| zone411 wrote:
| This was likely Copilot based on GPT 3.5.
|
| Microsoft: September 2022 to May 3rd, 2023
|
| Accenture: July 2023 to December 2023
|
| Anonymous Company: October 2023 to ?
|
| Copilot _Chat_ update to GPT-4 was Nov 30, 2023:
| https://github.blog/changelog/label/copilot/
| jjcm wrote:
| Great call out. I'd be extremely curious what the results
| looked like with something like cursor or just using claude out
| of the box. I'm amazed at just how easy it is to get simple
| small scripts up and going now with claude.
| agentultra wrote:
| Empirical studies like this are hard to conduct... I'm curious
| though. This study was authored by at least two folks _from
| Microsoft_ and one of the sample groups in the study was also
| _from Microsoft_. The reason this seems to stand out to me as odd
| is that Microsoft also owns the AI tool being used in the study
| and would definitely want a favourable conclusion in this paper.
|
| Is that a flag we should be watching out for?
| anonymousDan wrote:
| It sure is.
| gibsonf1 wrote:
| Yes, but what about all the bugs created and time to debug?
| croes wrote:
| >Notably, less experienced developers showed higher adoption
| rates and greater productivity gains.
|
| If you start low it's easier to get greater growth rates.
|
| The biggest is the first step, 0% to 1% is infinite growth.
| zombiwoof wrote:
| I sometimes wonder if LeetCode killed the Software Star
| MBCook wrote:
| It lets people make more PRs. Woohoo. Who cares?
|
| Does it increase the number of things that pass QA?
|
| Do the things done with AI assistance have fewer bugs caught
| after QA?
|
| Are they easier to extend or modify later? Or do they have rigid
| and inflexible designs?
|
| A tool that can help turn developers into unknown quality code
| monkeys is not something I'm looking for. I'm looking for a tool
| that helps developers find bugs or design flaws in what they're
| doing. Or maybe write well designed tests.
|
| Just counting PRs doesn't tell me anything useful. But it
| triggers my gut feeling that more code per unit time = lower
| average quality.
___________________________________________________________________
(page generated 2024-09-06 23:00 UTC)