[HN Gopher] New GitHub Copilot research finds 'downward pressure...
___________________________________________________________________
New GitHub Copilot research finds 'downward pressure on code
quality'
Author : ceejayoz
Score : 182 points
Date : 2024-01-28 18:02 UTC (4 hours ago)
(HTM) web link (visualstudiomagazine.com)
(TXT) w3m dump (visualstudiomagazine.com)
| ok123456 wrote:
| Code churn is up because the cost of trying new ideas is cheaper.
| Not all of the churn is because the code was defective.
| dangus wrote:
| The amount of things I'm expected to be able to accomplish
| constantly increases. I get it, I'm becoming more senior, but at
| the same time, it's kind of crazy that a single person can just
| be told to do the kind of things I'm told to do with the kind of
| deadlines I have. I can think of some takes I do by myself with
| code where a decade or two ago I'd be talking about hiring dozens
| of employees or contractors across the globe to physically enter
| a building and make a global business information system
| possible.
|
| I think this is related to the productivity wage gap. I'm being
| paid lower than I was in 2018 adjusted for inflation and I'm far
| more capable, producing far more value for my employer. As my
| tools get more valuable my labor gets proportionally less
| valuable. (A certain historical figure predicted this end state
| of capitalism).
|
| Here's another take: I don't give a fuck about the code quality
| at my company. I'm not paid extra to write good code. My bonus is
| based on basically nothing I can influence. If the whole company
| goes out of business on Monday I'll enjoy my long weekend.
| SoftTalker wrote:
| This is how everything goes.
|
| To build a house, it used to be necessary to hire experienced
| craftsmen. Now most of it is delivered pre-fabricated to the
| building site and laborers hired for the day nail it together.
|
| Cars used to be hand built by machinists and metalworkers. Now
| laborers tighten a bolt as cars roll by on an assembly line. Or
| a robot does it.
|
| Computer coding is going the same way. We lived and worked
| through the craftsman stage. Now we're becoming laborers.
| extheat wrote:
| This is why I am concerned that really good AI tools (be it LLMs,
| or not), are only going to make people dumber over time. It is
| true that those people that are highly motivated will be
| digitally enhanced by LLMs, but the hard reality is that this
| does not encompass most people. Most people will do the least
| amount of work possible to solve a task. Rather than taking the
| time to understand and critically reason about the output of an
| LLM, these people will instead take it and run, only asking
| questions later [if at all].
|
| And as we zoom out and think more farther into the future, I see
| it getting much worse. If AI is really doing all the "hard
| stuff", then the general human incentive to learn and do them at
| all quickly treads to 0. This isn't going to be everyone, I think
| some people will absolutely become "10x developers" or the
| equivalent for other domains. But all this will result in is more
| inequity, in my naive view. The universe has a fundamental
| property that things move from higher energy states into lower
| ones. From a human POV, I think you could apply a similar idea,
| if the need to be smart quickly recedes, then in general we may
| degrade unaugmented human collective intelligence over time.
|
| I don't know, maybe we'll figure something out to make it much
| easier to learn and ingest new concepts, but it seems more and
| more to me that the high obstacles for human brains learning
| things (with poor memory) is too big a bottleneck to overcome any
| time soon.
| badrequest wrote:
| "This tech will only make people dumber over time" was also
| claimed about the Internet.
| manvillej wrote:
| yes but in this case, we have actual research showing that
| the technology is degrading the quality of work.
| andersa wrote:
| Is that necessarily a bad thing? What if high quality isn't
| required to do the thing you wanted to accomplish?
|
| Many other mass produced products we buy today are clearly
| lower quality than ones crafted by artisans 50 years ago,
| and yet they do what we need them to do at a fraction of
| the cost.
| yoyohello13 wrote:
| These cheaper products are also far more wasteful and
| taxing on natural resources. This is the "growth at any
| cost" mindset. It's been great for us in the last 100
| years, but I'm not convinced it's actually sustainable in
| the long run.
| jwells89 wrote:
| I think that regardless of category, we should strive for
| making quality easier and cheaper to achieve and sustain.
| Unfortunately financial incentives favor velocity and
| volume at the cost of all else and so instead we're
| increasingly pumping out vast amounts of garbage.
| staunton wrote:
| > regardless of category
|
| I think that's a sweeping generalization. For example,
| it's much better to have a bunch of food that's mostly
| garbage than to have a famine where all food is super
| high quality. There were points in history where this
| choice was made (obviously unconsciously because the
| choice is too obvious to even think about). Other
| examples abound.
|
| Technology often makes things much cheaper while reducing
| quality. Sometimes that's bad, sometimes it's great.
| Attummm wrote:
| There is a threshold of quality; once it goes below, it's
| a broken product. Personally, I'd rather have a high-
| quality product crafted by artisans than mass-produced
| ones, similar to preferring a dinner at a restaurant made
| by a chef over fast food. However, in engineering, every
| choice involves tradeoffs.
| economicalidea wrote:
| We have research but I think it's far from rock-solid
| proven.
|
| Like any idea about squishy human brains and its products
| it remains to be seen and can't be as easily proven as for
| example research in Physics.
|
| I would say current research has a probability of 50% of
| being correct at best.
| bluedays wrote:
| Well, it wasn't wrong. The Internet _has_ made people dumber.
| pi-e-sigma wrote:
| Internet made us dumber but also made a lot of things
| easier. Just like industrial machines made us weaker (or
| rather, not as rugged as we used to be) but also allowed us
| to get much more work done.
| flal_ wrote:
| Well, to this day one can consider this true, looking at
| tiktok et al.
| jraph wrote:
| > one can consider this true
|
| Not without a solid result establishing so. But good luck
| with this, you'd probably need to compare our world with
| internet and a similar enough world but without it, which
| does not exist.
|
| In any case, internet can't be reduced to "tiktok et al"
| xanderlewis wrote:
| TikTok may indeed be a contributing factor to the current
| collective brain rot, but I've learnt more so far from
| YouTube and access to random PDF documents on any technical
| subject I can imagine than I probably would have done in
| one lifetime without.
| ptsneves wrote:
| It was claimed about writing as well. Oral experience passing
| was considered superior. The Quran puts great emphasis on
| decoration of it so it can be transmitted and recited orally.
| jazzyjackson wrote:
| same with phd defense
| staunton wrote:
| PhD defenses are mostly ceremonial. What are you trying
| to say? Is anything important transmitted orally during a
| PhD defense?
| tarboreus wrote:
| Are you sure it didn't?
| Karellen wrote:
| ...and about the written word, by Socrates
|
| https://blogs.ubc.ca/etec540sept13/2013/09/29/socrates-
| writi...
| pi-e-sigma wrote:
| In a sense Socrates was right. It's always better to be
| able to recall something from your memory rather than
| having to look it up in a book. The classic example is the
| multiplication table. The problem is that humans have
| rather limited capacity for such a 'recallable on demand'
| memory.
| staunton wrote:
| The problem is that you have to invest actual effort to
| memorize things. Memorizing things you very rarely need
| is not worth it, even if you're amazing at memorizing
| things. It's impossible to memorize everything you would
| benefit from having access to in written form.
| pi-e-sigma wrote:
| On the other hand just because something is written down
| somewhere is not at all equal to 'having access to it'.
| Once you forget it it might as well not exist at all. The
| information needs to be internalized to be any useful,
| and for that at least some of it needs to be kept in your
| memory
| staunton wrote:
| And that's why digital documents are better (for average
| use-cases) than paper. Another argument _for_ using
| technology to make things easier at the cost of possibly
| losing skills you no longer urgently need (such as
| "maintaining a library of paper books").
| sadjad wrote:
| This is a tale as old as time.
|
| In a conversation with Phaedrus, Socrates worries that
| writing could impair human memory, as people depend on
| written words instead of recalling information. [0]
|
| [0]: https://en.wikipedia.org/wiki/Phaedrus_(dialogue)#Discus
| sion...
| klipt wrote:
| It's true though, as human (+ tool) gets smarter, human
| (without tool) tends to get dumber in the domain the tool
| augments.
|
| The question is will we one day have tools so powerful,
| that the human is vestigial, and tool (without human) is
| just as powerful and cheaper than tool (+ human)?
| Ilverin wrote:
| This is the case today for ai chess engines
| staunton wrote:
| But there are still people playing chess professionally
| because playing well is not enough. (You have to
| entertain an audience, which is only possible by "just
| playing well" if you're human)
| jimbob45 wrote:
| True in general, yes, but writing is an elegant solution
| where the longer something is, the more likely you are to
| write it down because you are less likely to be able to
| remember it. The shorter something is, the less likely
| you are to write it down because it's a pain to get out
| your scratchpad (or iPhone) for ten words or less.
| klipt wrote:
| Yes writing can only augment human intelligence.
|
| AI might replace it.
| andybak wrote:
| I do think it's murkier then that.
|
| Writing "replaced" some forms of intellectual effort.
|
| And it's yet to be seen how AI will play out.
| williamcotton wrote:
| It's also an early example of the medium being the message.
| Socrates and his interlocutors are sometimes a parable on
| the transition from oral to written culture.
|
| I find that it is true that new mediums and new
| technologies for language have a numbing effect on certain
| senses and an amplification of others.
|
| Writing is beneficial in one regard but does have an impact
| on memory. Epic poems of great length were memorized in
| their entirety, a skill that would be a lot easier to
| develop in a world without writing.
| andybak wrote:
| Personally speaking, I've got much worse at stripping tree
| bark off with my teeth since I evolved opposable thumbs.
| Cheer2171 wrote:
| And they were right.
| joshuamcginnis wrote:
| We'd have to define "dumber" because adult literacy and
| numeracy rates have largely increased globally since the
| 1990's.
|
| https://ourworldindata.org/literacy
| klipt wrote:
| This is largely due to increasing access to education in
| less developed countries.
|
| In already developed countries the Flynn effect seems to be
| reversing, ie IQ is leveling off or even dropping.
| staunton wrote:
| IQ is a very different thing from literacy (or generally
| any skills one learns). I think IQ is not relevant to
| this discussion.
|
| It's basically impossible to improve an adult's g-factor,
| for example. For children, things like nutrition and
| hygiene (e.g. no parasites) play a big role. But the
| kinds of things a human learns or tools they use doesn't
| significantly affect G-factor.
| solardev wrote:
| Didn't it?
| ehutch79 wrote:
| And they seem to have been proven right. (probably not the
| way they think)
| wouldbecouldbe wrote:
| The mistakes that will be made are obvious, and those who
| fear that are correct, yet they will also be overcome because
| the ones not learning will fail and the ones learning will
| succeed.
| alangibson wrote:
| Whoever said it was right. One of the dumbest public figures
| of our time was elected president as a direct result of the
| reach and amplification provided by the internet.
|
| Qanon was purely a creation of the internet. Now go take a
| look at how many people believe one, many or all of the
| various Qanon alternative facts.
| jraph wrote:
| Those are convincing arguments that internet has (major)
| downsides and bad consequences. The scale of surveillance
| it enables is another one.
|
| They don't prove that it makes people dumber. You have to
| quantify and qualify people and to define "dumb".
|
| Maybe people are not actually dumber because of the
| internet, but the internet is very good at spreading ideas,
| including incredibly dumb ones, especially (because of how
| human beings work) those likely to cause feeling of
| outrage.
|
| Maybe people are not dumber, just too defenseless against
| the scale of bullshit they are faced with because of the
| internet. Maybe internet is an incredibly good tool, but
| strongly requires good learning / training of critical
| thinking and there's not enough of this yet.
| staunton wrote:
| People used to believe _a lot_ of crazy conspiracies and
| superstitions throughout history. The reason why we 're so
| appalled at Qanon is that "it should be _so easy_ for them
| to correct their superstitions, given the tools we have ".
|
| It's hard to argue that nowadays people believe more crazy
| stuff than before the internet was invented. (It's very
| easy, of course, to _claim_ this, as many like to do.)
| xanderlewis wrote:
| Yes. It seems that the proportion of people for whom 50%
| of their beliefs constitute unfounded, unjustified
| nonsense is actually far lower than it was before the
| internet. One of the main things that has changed,
| though, is how _quickly_ ideas (including bonkers ones)
| spread. So whereas we used to see a significantly new
| bonkers idea only every few centuries, we seemingly now
| see several a week.
| noobermin wrote:
| It did.
| jrockway wrote:
| This sounds pretty similar to all technology. I remember
| Slashdot threads about how IntelliSense was making software
| engineers dumber. I remember my teachers saying that allowing
| calculators on the SAT made kids dumber. I remember people
| saying that spell checkers would make people worse at writing.
| I remember people saying at the start of the pandemic that Zoom
| meetings would make everyone anti-social.
|
| None of this really happened. I mostly use editor tools to
| automate away tedium that doesn't matter; I type "log.Inf<TAB>"
| and it adds "src/internal/log" as an import at the top of the
| file and types the o and open parenthesis for me. I have not
| forgotten how to do that myself, but it saves me a couple
| seconds. Calculators didn't really make people dumber, though I
| have to say that a lot of arithmetic I learned in elementary
| school did make me dumber ("touch math" was the killer for me;
| slows me down every time I do arithmetic in my head; I need
| some brainwashing program that deletes that from my brain).
| Spell check didn't make people worse at spelling; spelling
| things wrong still has a penalty (C-w to kill the last word and
| spell it correctly), so the incentive is to still to lurn how
| to spel wrds rite. Zoom meetings didn't ruin the corporate
| world; I personally found them very helpful for memorizing
| people's names with a high degree of certainty. You see it
| under their face for 40 minutes at a time, and you learn it
| fast. In real life, probably takes me a few weeks for people I
| only see once a week. So, honestly a benefit for me.
|
| The current state of AI seems very similar to these
| technologies. I did a copilot free trial (and didn't renew it).
| With the free trial I think there were a couple things it was
| good at. One time I wanted a CPU profile for my app, so I just
| asked the AI to type it in. Open a file, check for an error,
| start profiling to the file, stop profiling, close the file.
| Would have taken me a minute or two to type in, but Copilot
| typed it in instantly. I also did something like "do the same
| as the function above, but for a YAML file instead of a JSON
| file". Again, super trivial to type in that code, it's really
| only one line, but the AI can handle that just fine in an
| instant. I don't really think it's more than slightly smarter
| IntelliSense, but without any access to the compiler toolchain,
| so it can sometimes just hallucinate stuff that doesn't exist.
|
| I've found this to be kind of an interesting way to proofread
| documents and design APIs. Give ChatGPT a document you're
| working on, and then ask questions about it. If it gets the
| wrong answer, then your doc is no good! Similarly, ask it how
| to write some code using your new API (without showing it the
| API). Whatever it writes should be the surface area of your
| API. This avenue is pretty interesting to me, and it's not
| replacing humans, it's just a smarter rubber duckie.
|
| Overall, I think we're in a little bit of a hype phase with AI.
| I look at it kind of like a dog that has read every book, if
| such a thing were possible. Pretty smart, but not quite human
| yet. As a result, it's not going to do well in the areas where
| people really want to apply it; customer service, insurance
| claims, loan underwriting, etc. But it is pretty good for
| asking questions like "does my document make sense" or "please
| find some boilerplate to cut-n-paste here, I am going to delete
| this before checking in anyway". Also not too bad at slightly
| modifying copyrighted images ;)
| lolinder wrote:
| In Phaedrus, Plato has Socrates tell the story of a dialog
| between Egyptian gods as a caution against writing--that
| writing would cause people to lose their ability to remember.
| On the one hand, Thamus's warning was accurate--cultures that
| rely on writing generally do not have robust memorized oral
| traditions--but on the other hand, we only have this story
| today because Plato wrote it down, and he cannot have been
| ignorant of the irony.
|
| Every tool has this trade-off, and the _existence_ of skills
| that will be lost is not evidence that the tool will do more
| harm than good. I don 't think anyone here would argue that
| Socrates was correct that writing would be the end of memory
| and wisdom.
|
| > To [Thamus] came Theuth and showed his inventions ... when
| they came to letters, "This,* said Theuth, "will make the
| Egyptians wiser and give them better memories; it is a specific
| both for the memory and for the wit."
|
| > Thamus replied: "O most ingenious Theuth, the parent or
| inventor of an art is not always the best judge of the utility
| or inutility of his own inventions to the users of them. And in
| this instance, you who are the father of letters, from a
| paternal love of your own children have been led to attribute
| to them a quality which they cannot have; for this discovery of
| yours will create forgetfulness in the learners' souls, because
| they will not use their memories; they will trust to the
| external written characters and not remember of themselves. The
| specific which you have discovered is an aid not to memory, but
| to reminiscence, and you give your disciples not truth, but
| only the semblance of truth; they will be hearers of many
| things and will have learned nothing; they will appear to be
| omniscient and will generally know nothing; they will be
| tiresome company, having the show of wisdom without the
| reality."
|
| https://www.gutenberg.org/files/1636/1636-h/1636-h.htm
| mrweasel wrote:
| People have different workflows, but mine is frequently, skim the
| documentation, make a prototype, refine code a bit, add tests,
| move stuff around, break stuff, rework code, study documentation,
| refactor a bit more, and then at that point I have enough
| understanding of the problem to go in at yank out 80% of my code
| and do it right.
|
| If Copilot gives me working code in the prototype stage, good
| enough that I can just move on to the next thing, my
| understanding is never going to be good enough that I can go in
| and structure everything correctly. It will effectively allow me
| to skip 90% of my workflow, but pay the price. That's not to say
| that Copilot can't be extremely helpful during the final steps of
| development.
|
| If those findings are correct, I can't say that I'm surprised.
| Bad code is written by poor understanding and Copilot can't have
| any understanding beyond what you provide it. It may write better
| code than the average programmer, but the result is no better
| than the input given. People are extremely focused on "prompt
| engineering", so why act surprised when a poor "prompt" in VScode
| yields a poor result?
| andybak wrote:
| I'm not sure why you decided that "use copilot" also implies
| missing out most of your later steps. Who decides to skip all
| those steps? Presumably you?
|
| My experience is that Copilot is great at getting me started.
| Sometimes the code is good, sometimes it's mediocre or
| completely broken.
|
| But it's invaluable at getting me thinking. I wasted a lot more
| time before I started using it. That might just be my weird
| brain wiring...
|
| (Edited to sound less narky. I shouldn't post from a mobile
| device)
| fwsgonzo wrote:
| I recently tried Copilot out of curiosity and this is my
| experience too: It helps me getting started, which for me is
| 99% of the challenge. I know how to solve problems, even
| complex ones, but for some reason getting started is just so
| extremely hard, sometimes.
|
| Copilot lets me get started, even if it's wrong sometimes.
| There have been times where I have been surprised by how it
| took something I wrote for a server, and presented the
| correct client-side implementation.
|
| I've used it a few times to describe a problem and let it
| handle the solution. It's not very good, but I wonder if one
| should place more blame on PEBCAK and put more time into
| problem-description. I gave it a few more paragraphs to
| describe the problem, and eventually I could take it from
| there. It was still wrong, but enough to get me started.
| Immensely helpful that way.
|
| Another aspect that I'm wondering about is if it will be able
| to do more with better documented code. Anyone have
| experience with that? I've started to write more doxygen
| comments, and hoping to see if there's a slow shift to more
| accurate predictions.
| simonw wrote:
| The full paper is here: https://gitclear-public.s3.us-
| west-2.amazonaws.com/Coding-on...
| stusmall wrote:
| Thank you for the source.
| padjo wrote:
| The real news here is that the authors have apparently found an
| objective and universal measure of code quality.
| o11c wrote:
| Eh, it's not that such measures don't exist - they're just
| noisy.
|
| And the thing about AI is that their negative impact is clearly
| visible above the noise.
| godzillabrennus wrote:
| I decided to use ChatGPT to build a clone of Yourls using
| Django/Python. I gave it specific instructions to not only allow
| for a custom shortened URL but to track the traffic. It didn't
| properly contemplate how to do that in the logic or data model. I
| had to feed it specific instructions afterwards to get it fixed.
|
| AI tools are akin to having a junior developer working for you.
| Except they are much much faster.
|
| If you don't know what you're doing they just accelerate the pace
| that you make mistakes.
| KronisLV wrote:
| > AI tools are akin to having a junior developer working for
| you. Except they are much much faster.
|
| Honestly, this is brilliant. The other day I had to add table
| name prefixes to a SELECT statement column aliases, since such
| a feature just doesn't exist for some reason, a bit like:
| -- fails because of duplicate column names (e.g. when creating
| a view) SELECT * FROM table_a JOIN
| table_b ON ... JOIN table_c ON ... ...
| -- this would solve my issue, if WITH_PREFIX did exist (or
| anything like it) SELECT table_a.* WITH_PREFIX
| 'table_a', table_b.* WITH_PREFIX 'table_b',
| table_c.* WITH_PREFIX 'table_c' FROM table_a JOIN
| table_b ON ... JOIN table_c ON ... ...
|
| So I just gave ChatGPT the schema definitions/query and it
| wrote out the long list of like 40 columns to be selected for
| me, like: SELECT table_a.id AS
| 'table_a_id', table_a.email AS 'table_a_email',
| ... table_b.id AS 'table_b_id',
| table_b.start_date AS 'table_b_start_date', ...
|
| and so on. I haven't found another good way to automate things
| like that across different RDBMSes (different queries for
| system tables that have schema information) and while it's
| possible with regex or a bit of other types of text
| manipulation, just describing the problem and getting the
| output I needed was delightfully simple.
|
| Aside from that, I just use the LLMs as autocomplete, which
| also encourages me to have good function naming, since often
| enough that's sufficient information for the LLM to get started
| with giving me a reasonable starting point. In particular, when
| it comes to APIs or languages I haven't used a lot, but the
| problems that I face have been solved by others thousands of
| times before. I don't even have to use StackOverflow much
| anymore.
|
| That's why I bought Copilot (though JS/HTML autocomplete in
| JetBrains IDEs is visually buggy for some reason) and use
| ChatGPT quite a lot.
|
| LLMs are definitely one of my favorite things, after
| IntelliSense (and other decent autocomplete), codegen (creating
| OpenAPI specs from your controllers, or bootstrapping your
| EF/JPA code from a live dev database schema), as well as model
| driven development (generating your DB schema migrations/tables
| from an ER model) and containers (easily packaged, self-
| contained environments/apps) and smart IDEs (JetBrains ones).
| konschubert wrote:
| > If you don't know what you're doing they just accelerate the
| pace that you make mistakes.
|
| 100%
|
| And if you know what you are doing, they will accelerate the
| way you're building stuff.
| geraneum wrote:
| It's not always clear to everyone that there's something they
| don't know!
| cleandreams wrote:
| My question is, how do you become a senior developer when the
| junior developer just keeps throwing "working" "good enough"
| code at you?
|
| I think companies will want more code faster to the extent that
| fewer people will emerge from the churn really knowing what
| they are doing.
| ryanmcgarvey wrote:
| Maybe it's worth reevaluating our definition of quality?
|
| In a world where AI can read our codebase, ingest a prompt, and
| quickly output "correct" if not clean and concise code, and then
| be able to iterate on code with more prompt, do we need all the
| same patterns we used to adopt when humans were painstakingly
| writing every line of code?
|
| This reminds of of the CISC to RISC migration - now that
| computers are in the loop writing the tedious parts, we don't
| need to burden our codebase with patterns meant to relieve humans
| from the tedium.
|
| I find myself, for instance, writing more long form, boring
| configuration files that once upon a time I would have built some
| abstraction reduce the boilerplate and verbosity. But now that
| co-pilot can just auto-complete the next section for me, why
| bother?
| noobermin wrote:
| https://en.wikipedia.org/wiki/Wirth%27s_law
| arp242 wrote:
| > In a world where AI can read our codebase, ingest a prompt,
| and quickly output "correct" if not clean and concise code, and
| then be able to iterate on code with more prompt, do we need
| all the same patterns we used to adopt when humans were
| painstakingly writing every line of code?
|
| That world does not exist, so currently this line of thinking
| is academic.
|
| Perhaps it will exist in the future, but it's far from a
| certainty if that will come to pass, and unclear on what kind
| of time-frame. Personally I'm quite skeptical any of us will
| see it within our lifetimes.
| patrickmay wrote:
| Patterns aren't intended to "relieve humans from the tedium,"
| they're to make the code more intelligible. Code generators
| create notoriously difficult to understand code. Apparently the
| current crop of LLMs are no better in that regard.
|
| > find myself, for instance, writing more long form, boring
| configuration files that once upon a time I would have built
| some abstraction reduce the boilerplate and verbosity. But now
| that co-pilot can just auto-complete the next section for me,
| why bother?
|
| Again, so humans can understand it more easily.
| spaniard89277 wrote:
| I'm a junior, and I have Codeium installed in VSCode. I've found
| it very distracting most of the times, I don't really understand
| why so many people uses this kind of assistants.
|
| I find stuff like Phind useful, in the sense that sometimes
| something happens that I don't understand, and 60% of the times
| Phind actually helps me to understand the problem. Like finding
| trivial bugs that I didn't spot because I'm tired, dumb, etc.
|
| On the other hand, with Codeium, I guess it may be useful when
| you're just churning boilerplate code for some framework, but in
| my little expericence (writing scrapers and stupid data pipelines
| & vanilla JS + HTML/CSS) cycling through suggestions is very
| irritating, specially because many times it doesn't work. Most of
| the times for stupid reasons, like lacking an argument or
| something like that, but then it's time you have to spend
| debugging it.
|
| Another problem I have is that I find there's a common style of
| JS which consist in daisy-chaining a myriad of methods and
| anonymous functions, and I really struggle with this. I like to
| break stuff into lines, name my functions and variables, etc. And
| so many times code suggestions follow this style. I guess it's
| what they've been trained on.
|
| Codeium is supposed to learn from this, and sometimes it does, to
| be fair.
|
| But what I worry the most is that, If I'm a junior and I let this
| assistants do the code for me ?How the hell I'm supposed to
| learn? Because giving Phind context + questions helps me learn or
| gives me direction to go on find it by myself in the internet,
| but if the only thing I do is press tab, I don't know how the
| hell I'm supposed to learn.
|
| I found a couple days ago that many people (including devs) are
| not using LLMs to get better but it's just a substitute of their
| effort. Isn't people afraid of this? Not because companies are
| going to replace you, but it's also a self-reflection issue.
|
| Coding is not the passion of my life, addmitedly, but I like it.
| I like it because it helps me to make stuff happen and to handle
| complexity. If you can't understand what's happening you won't be
| able to make stuff happen and much less to spot when is
| complexity going to eat you.
| withinboredom wrote:
| I think probably the best use of AI, so far, was when I went
| into a controller and told it to generate an openAPI spec ...
| and it got it nearly right. I only had to modify some of the
| models to reflect reality.
|
| BUT (and this is key), I've hand-written so many API specs in
| my career that 1) I was able to spot the issues immediately,
| and 2) I could correct them without any further assistance
| (refining my prompt would have taken longer than simply fixing
| the models by hand).
|
| For stuff where you know the domain quite well, it's amazing to
| watch something get done in 30s that you know would have taken
| you the entire morning. I get what you're saying though, I
| wouldn't consider asking the AI to do something I don't know
| how to do, though I do have many conversations with the AI
| about what I'm working on. Various things about trade-offs,
| potential security issues, etc. It's like having a junior
| engineer who has a PHD in how my language works. It doesn't
| understand much, but what it does understand, it appears to
| understand it deeply.
| staunton wrote:
| > I wouldn't consider asking the AI to do something I don't
| know how to do
|
| My experience has been the opposite so far. I benefit much
| more from such tools when I can _easily check if something
| works correctly_ and would have to learn /look up a lot of
| easy and elementary stuff to do it from scratch.
|
| For example, adding to some existing code in a language I
| don't know and don't have time or need to learn (I guess not
| many people are often in that situation). I get a lot of
| hints for what methods and libraries are available, I don't
| have to know the language syntax, for easy few-line snippets
| (that do standard things and which I can test separately) the
| first solution usually just works. This is deliberately
| passing on an opportunity for deeper and faster learning,
| which is a bad idea in general, but sometimes the speed
| trade-off is worth it.
|
| On the other hand, for problems where I know how to solve
| them, getting some model to generate the solution I want (or
| at least one I'm happy with) tends to be more work than just
| doing it myself.
|
| I probably could improve a lot in how I use the available
| tools. Haven't had that much opportunity yet to play with
| them...
| jacquesm wrote:
| > Coding is not the passion of my life, addmitedly, but I like
| it.
|
| It may not be the passion of your life but I haven't seen
| anybody articulate better (in recent memory) what they want to
| get out of coding and how they evaluate their tools. Keep at
| it, don't change and you'll go places, you are definitely on
| the right path.
| tpmoney wrote:
| The tool and design of the tool matters a lot. I've used
| Codeium in VSC and GH Copilot in Intellij, and the experience
| (and quality) of the GH + Intellij paring is much better than
| Codeium + VSC.
|
| My biggest use for AI assistants has been speeding up test
| writing and any "this but slightly different" repetitive
| changes to a code base (which admittedly is also a lot of test
| writing). At least in intellij + GH, things like, a new
| parameter that now needs to be accounted for across multiple
| methods and files is usually a matter of "enter + tab" after
| I've manually typed out the first two or three variants of what
| I'm trying to do. Context gives it the rest.
|
| In VSC with Codeium, the AI doesn't seem quite as up to snuff,
| and the plugin is written in such a way that its suggestions
| and the keys for accepting them seem to get in the way a lot.
| It's still helpful for repetitive stuff, but less so for
| providing a way of accomplishing a given goal.
| mvdtnz wrote:
| > Another problem I have is that I find there's a common style
| of JS which consist in daisy-chaining a myriad of methods and
| anonymous functions, and I really struggle with this. I like to
| break stuff into lines, name my functions and variables, etc.
|
| I think your whole comment is excellent but I just wanted to
| tell you, you're on the right track here. Certain developers,
| and in particular JS developers, love to chain things together
| for no benefit other than keeping it on one line. Which is no
| benefit at all. Keep doing what you're doing and don't let this
| moronic idiom infect your mind.
| xanderlewis wrote:
| Sometimes making something a one-liner is itself a benefit
| for readability. Especially if you're used to reading it. But
| admittedly it's very easy (and can be tempting) to take it
| too far...
| dweinus wrote:
| The methodology seems to be: compare commit activity from 2023 to
| prior years, without any idea of how many involve Copilot. Then
| interpret those changes with assumptions. That seems a bit
| shakey.
|
| Also: "The projections for 2024 utilize OpenAI's
| gpt-4-1106-preview Assistant to run a quadratic regression on
| existing data." ...am I to understand they asked gpt to do a
| regression on the data (4 numbers) rather than running a simple
| regression tool (sklearn, r, even excel can do this)? Even if
| done correctly, it is not very compelling when based off of 4
| data points and accounting for my first concern.
| zemo wrote:
| check out the paper, not just the summary. They explain their
| methodology. The output has four data points because it's a
| summary. The input is ... more data than that.
| lolinder wrote:
| More data, but OP is right on the weaknesses of the study--
| the author posted here [0] and acknowledged that they can't
| actually say anything about causality, just that 2023 looked
| different than 2020.
|
| [0] https://news.ycombinator.com/item?id=39168841
| graypegg wrote:
| Feels like taking the average of all text on the internet, does
| result in barely OK output. Very few people are truly "average"
| in the numerical sense of the word, so you're going to see better
| results from most devs compared to the coefficient soup that
| tries to find the commonalities between the best and worst devs.
|
| Truth be told though, speaking as someone that still does not use
| LLM tools at work... "just OK" is totally viable for a lot of
| things. Prototypes yes, products expected to be around a while,
| maybe not.
| withinboredom wrote:
| I recently had this discussion at work. There's a difference
| between writing software that will be seen again in a few
| months and writing software that probably will never be touched
| again (or at least touching it again will be pretty risky).
| Identifying which one you're working on is super important.
|
| When you're writing software that is touched fairly often, the
| "tribal knowledge" of how it works will likely live-on. You can
| be a little bit clever at times, you don't need to comment as
| heavily, and your variable names can be a bit wonky.
|
| When you're writing software that is hardly ever touched ...
| everything needs to be crystal clear. Write lots of comments,
| explaining the "why" things are the way they are. You want to
| avoid cleverness except in the name of performance (but then
| comment how you expect it to work so that when someone comes
| along to change it, they understand wtf you are doing and the
| constraints you used to define the code). It's a totally
| different ballgame.
|
| AI doesn't get this distinction, hell, most programmers don't
| either.
| bemusedthrow75 wrote:
| > "the disjointed work of a short-term contractor"
|
| Speaking as someone who has been a short-term contractor,
|
| 1) the issue with short-term contractors is not the work, it's
| the project management -- so in this analogy, the project manager
| is the idiot who thinks using Copilot to do just the bits they
| don't understand or don't want to do is going to improve things
|
| 2) go f### yourself
|
| Yes, this is disrespectful, but no more derogatory than
| associating "disjointed work" with an arbitrary group, or the
| study talking of "itinerant" contributors. These people
| contribute because they will write code nobody else would or has;
| if the code has not subsequently been "made more DRY" or better-
| integrated into the codebase, it isn't the fault of the
| contributor, it's the fault of the person managing the codebase.
|
| Short-term contributors very often find themselves being tasked
| to maintain code that nobody else will, and limited in the scope
| of what can improve through project management, budget or time
| restrictions.
|
| It is in general asinine to talk of the work of _a_ (single)
| contractor being "disjointed", when it is the whole that is
| disjointed, and it contributes to the blame culture around
| contractors.
|
| I once took on a project where it was very clear just from code
| styles that three short-term contractors had worked on it. Were
| any of them individually to blame for it not working? It became
| very clear, after dealing with the client, that they were not.
|
| To be clear: I think using Copilot is a lazy, expedient crutch
| that won't go anywhere good. But the framing of this study
| suggests that the magic is in the "senior developer"; it's not.
| It's in the ownership.
| wowspeechless wrote:
| You're absolutely right. Why are they maligning contractors?
| Never mind the fact that the comment was insulting - it's not
| even factual. I'm hired as a contractor to fix broken projects.
| klyrs wrote:
| Garbage in, garbage garbage garb garb ggggggggggg
| wbharding wrote:
| Original research author here. It's exciting to find so many
| thinking about long-term code quality! The 2023 increase in
| churned & duplicated (aka copy/pasted) code, alongside the
| reduction in moved code, was certainly beyond what we expected to
| find.
|
| We hope it leads dev teams, and AI Assistant builders, to adopt
| measurement & incentives that promote reused code over newly
| added code. Especially for those poor teams whose managers think
| LoC should be a component of performance evaluations (around 1 in
| 3, according to GH research), the current generation of code
| assistants make it dangerously easy to hit tab, commit, and seed
| future tech debt. As Adam Tornhill eloquently put it on Twitter,
| "the main challenge with AI assisted programming is that it
| becomes so easy to generate a lot of code that shouldn't have
| been written in the first place."
|
| That said, our research significance is currently limited in that
| it does not directly measure what code was AI-authored -- it only
| charts the correlation between code quality over the last 4 years
| and the proliferation of AI Assistants. We hope GitHub (or other
| AI Assistant companies) will consider partnering with us on
| follow-up research to directly measure code quality differences
| in code that is "completely AI suggested," "AI suggested with
| human change," and "written from scratch." We would also like the
| next iteration of our research to directly measure how bug
| frequency is changing with AI usage. If anyone has other ideas
| for what they'd like to see measured, we welcome suggestions! We
| endeavor to publish a new research paper every ~2 months.
| lolinder wrote:
| > That said, our research significance is currently limited in
| that it does not directly measure what code was AI-authored --
| it only charts the correlation between code quality over the
| last 4 years and the proliferation of AI Assistants
|
| So, would a more accurate title for this be "New research shows
| code quality has declined over the last four years"? Did you do
| anything to control for other possible explanations, like the
| changing tech economy?
| oooyay wrote:
| > We hope it leads dev teams, and AI Assistant builders, to
| adopt measurement & incentives that promote reused code over
| newly added code.
|
| imo, this is just replacing one silly measure with another.
| Code reuse can be powerful _within_ a code base but I 've
| witnessed it cause chaos when it spans code bases. That's to
| say, it can be both useful and inappropriate/chaotic and the
| result largely depends on judgement.
|
| I'd rather us start grading developers based on the _outcomes_
| of software. For instance, their organizational impact compared
| to their resource footprint or errors generated by a service
| that are not derivative of a dependent service /infra. A
| programmer is responsible for much more than just they code
| they right; the modern programmer is a purposefully bastardized
| amalgamation of:
|
| - Quality Engineer / Tester
|
| - Technical Product Manager
|
| - Project Manager
|
| - Programmer
|
| - Performance Engineer
|
| - Infrastructure Engineer
|
| Edit: Not to say anything of your research; I'm glad there are
| people who care so deeply about code quality. I just think we
| should be thinking about how to grade a bit differently.
| axegon_ wrote:
| I'm not surprised. AI tools can be great at providing a quick,
| working example in simple scenarios, but it's just that: a
| quick(often dirty) working example. But I've seen people taking
| it as-is and putting it into a production codebase. Sure, a
| function that iterates over an array, and checks if some item
| exists - fine. In those cases it will do the job simply because
| you(it) can't get it wrong. However I had this experience where
| senior developers were fully invested into using it. And because
| managers see code just erupting like a volcano, they embrace it
| and in fact rise their expectation when it comes to delivering a
| feature: "you could do it in a week before, you should be able to
| do it in 2 hours now, right?". And on more than one occasion this
| has backfired massively in everyone's face. The last time me and
| another dev spent two straight days rewriting and debugging
| everything cause there was an international exhibition that was
| about to start and the company was at the front of the line and
| everyone else simply pushed a ton of code that was 75% AI-
| generated, completely ignoring the edge cases, which were more
| than people anticipated.
|
| But probably the most off-putting thing I've experienced is an
| OKR session with 50 people in it, where a lead dev publicly
| opened chatgpt, prompted "how do we reduce the number of critical
| bugs by 30% in the next quarter", chatgpt came up with a generic
| response, everyone said "perfect", copy-paste that into Jira and
| call it a day. And I'm just sitting there and wondering if there
| was something rotten in my breakfast and I'm hallucinating.
| Unfortunately my breakfast was fine and that really happened. The
| few times I've tried using those, they were only helpful with
| dumb, repetitive and mundane tasks. Anything else, you have to
| rinse and repeat until you get a working solution. And when you
| read it(for those of us that do), you realize you might have been
| better off recruiting a freelancer from year one in University to
| spend a day mashing it up and likely coming up with something
| better.
|
| But I bet much of those year ones would be doing this exact thing
| day in and day out until they come up with a solution:
| Occasionally I will grab my laptop and go work at a cafe on my
| personal projects for a change and I can't tell you how many
| times I've seen people just copy pasting stuff from chatgpt and
| pasting it back into their IDE/editor and calling it a day -
| students and clearly people who are doing this for a living. Not
| to mention copilot, that's the de-facto standard at this point.
|
| In fact I had this conversation last year with a guy(developer)
| who is 20-something years older than me(so mid 50-s): most of the
| LLM's are trained on stuff that is in the documentation,
| examples, reddit and stackoverflow. It's only a question of time
| until the content found in those exact locations where the
| training data is pulled from will become more and more AI-
| generated, models will be re-trained on those and eventually shit
| hitting the fan. I don't think we are too far off from this
| event.
| staunton wrote:
| > chatgpt came up with a generic response, everyone said
| "perfect", copy-paste that into Jira and call it a day
|
| I don't think I can believe this story as told...
| axegon_ wrote:
| 5 people(myself included) announced we resign the following
| day, so you better believe it.
| gnabgib wrote:
| Previously (same URL): [0](5 points, 2 days ago, 5 comments)
| [2](17 points) [3](9 points, 3 comments)
|
| "Poor code quality due to AI assistants GitHub Copilot and
| ChatGPT" [1](21 points, 2 days ago, 10 comments)
|
| [0]: https://news.ycombinator.com/item?id=39142285 [1]:
| https://news.ycombinator.com/item?id=39144366 [2]:
| https://news.ycombinator.com/item?id=39156643 [3]:
| https://news.ycombinator.com/item?id=39164079
| darepublic wrote:
| Sometimes less dry code can actually be easier to read and
| understand at the point of usage than dry code that has been more
| highly abstracted and requires grokking a sort of quasi DSL that
| defines the abstraction. Assuming that AI contributions will only
| increase, if a codebase were almost completely written by AI
| perhaps the benefits of DRY would diminish vs on the spot
| readability by humans only trying to understand the code and not
| personally add to it
| chongli wrote:
| Well as with anything, DSLs are subject to the rules of good
| design. A really well-designed DSL (such as SQL) takes on a
| life of its own and becomes incredibly ubiquitous and useful to
| know in its own right. Many other DSLs are totally unknown, not
| worth learning, and serve as barriers to code understanding.
|
| I don't know of too many people who would advocate replacing
| SQL with hand-written C manipulating B-trees and hash tables.
| Similarly, it's pretty rare that you want to hand-roll a parser
| in C over using a parser generator DSL or even something like
| regex.
| rsynnott wrote:
| Shock horror, the thing everyone said would happen has happened.
| I mean, I'm not sure what people expected.
| andybak wrote:
| Some people are thinking hard about a complex, nuanced topic of
| which we have very little past experience to draw on. I'm glad
| the conclusion is so self-evident to you. I must be a little
| slow.
| righthand wrote:
| ML/LLMs are nuanced and complex topics as they are the inner
| workings of automation. People using ML/LLM to get around
| having to write code already understood well enough isn't a
| complex nuanced topic because it is the outer workings of
| automation and has been studied in other fields quite
| extensively. No one should be surprised at the trend of
| lazier development from wide adoption of automation tools.
| andybak wrote:
| How do past lessons about automation cleanly apply to AI
| code generation?
|
| Cleanly enough that there's no room for debate, doubt or
| discussion?
|
| (Edit. Not the same person. I keep making this mistake in
| discussion threads)
|
| Previously I wrote:
|
| > It's your tone of "this is obvious, people! Why are you
| still wasting time thinking about it?" that I'm taking
| exception to.
| danielovichdk wrote:
| When I look into the future, and I know that I really can't, one
| thing I really believe in is that there will be a shift in how
| quality will be perceived.
|
| With all things around me there is a sense that technology is to
| be a saviour for many very important things - ev's, medicine, it,
| finance etc.
|
| At the same time it is more and more clear to me that technology
| is used primarily to grow a market, government, country etc. But
| it does that by layering on top of already leaking abstractions.
| It's like solving a problem by only trying to solvent be its
| symptoms.
|
| Quality has a sense of slowness to it which I believe will be a
| necessary feat, both due to the fact that curing symptoms will
| fall short and because I believe that the human species simply
| cannot cope with the challenges by constantly applying more
| abstractions.
|
| The notion about going faster is wrong to me, mostly because I as
| a human being do not believe that quality is done by not
| understanding the fundamentals of a challenge, and by trying to
| solve it for superficial gains is simply unintelligent.
|
| LLMs is a disaster to our field because it caters to the average
| human fallacy of wanting to reach a goal but without putting in
| the real work to do so.
|
| The real work is of course to understand what it is that you are
| really trying to solve with applying assumptions about
| correctness.
|
| Luckily not all of us is trying to move faster but instead we are
| sharpening our minds and tools while we keep re-learing the
| fundamentals and applying thoughtful decisions in hope to make
| quality that will stand the test of time.
| jstummbillig wrote:
| > The real work is of course to understand what it is that you
| are really trying to solve with applying assumptions about
| correctness.
|
| In how far do you think LLMs stand in the way of that?
|
| My experience has been very much the opposite: Instead of
| holding the hard part of the process up by digging through
| messy apis or libraries, LLMs (at least, in their current form
| but I suspect that this will theoretically simply remain true)
| make it painfully obvious when my thinking about a task of any
| significance is not sound.
|
| To get anywhere with a LLM, you need to write. To write, you
| have to think.
|
| Very often I find the most beneficial part of the LLM-coding-
| process is a blended chat backlog that I can refer back to,
| consisting of me carefully phrasing what it is that I want to
| do, being poked by a LLM, and me through this process finding
| gaps and clarifying my thoughts at the same time.
|
| I find this tremendously useful, specially when shaping the app
| early, to keep track of what I thought needed to be done and
| then later being able to reconsider if that is actually still
| the case.
| peterashford wrote:
| This aligns with my thinking about the utility of LLMs:
| they're rubber duck programming as a service
| chiefalchemist wrote:
| > LLMs is a disaster to our field because it caters to the
| average human fallacy of wanting to reach a goal but without
| putting in the real work to do so.
|
| It's a tool. It doesn't make sense to blame the tool. Is it the
| screwdriver's fault it gets used as a hammer? Or a murder
| weapon?
|
| Used intelligently Copilot & Co can help. It can handle the
| boilerplate, the mundane and free up the human element to focus
| on the heavy lifting.
|
| All that aside, it's early days. It's too early to pass
| judgement. And it seems unlikely it's going to go away.
| norir wrote:
| There is an interview with the great jazz pianist Bill Evans
| (conducted by his brother) in which he muses that most amateur
| musicians make the mistake of overplaying. They go out to the
| club and hear a professional and they come home and try to
| approximate what the professional does. What they end up with
| is a confused mess with no foundation. He insists that you have
| to learn to be satisfied with doing the simple things and
| gradually building up a stronger foundation.
|
| I think his insight applies nearly as well to using code
| generated by an ai.
| kvonhorn wrote:
| Quality is best thought of as a process, and that process got
| pushed out of the SDLC by Agile process and its metric of
| velocity. The use of LLM-generated code to further increase
| velocity in the absence of quality process is an obvious result.
| cleandreams wrote:
| I am worried that AI assisted code will be a competitive
| advantage so that the downsides will not be addressed until there
| are serious failures in critical code. Boeing but for power
| plants, for example.
| mvdtnz wrote:
| There was already a backlash against DRY code occurring before
| "AI" assistants hit the market, sadly. It was a growing movement
| when I was using Twitter in 2019-2022.
|
| Some younger developers have a very different attitude to code
| than what I was brought up with. They have immense disdain for
| the Gang of Four and their design patterns (probably without
| realising that their favourite frameworks are packed to the gills
| with these very patterns). They speak snidely about principles
| like DRY and especially SOLID. And on places like Twitter the
| more snide and contrarian you can be, the more engagement you'll
| get. Very disturbing stuff.
| Kiro wrote:
| Yeah, like this dude:
| https://twitter.com/ID_AA_Carmack/status/753745532619665408
| mvdtnz wrote:
| "DRY isn't absolute" is not what I was talking about. No
| serious developer considers any principle "absolute".
| vasco wrote:
| Only a sith deals in absolutes.
| ukuina wrote:
| Don't try it, he has the high ground.
| deadbabe wrote:
| DRY is mostly bad. You couple a lot of things together by
| making them all share the same code and then a small change to
| that code breaks various unrelated components.
| abathur wrote:
| I don't think it's mostly good or mostly bad in the abstract.
|
| DRYing code repeated for the same reason is mostly good.
|
| DRYing code coincidentally repeated for different reasons
| will sow code churn or inadvertent behavior shifts.
| charles_f wrote:
| Dry's mostly a smell test to be applied when copying stuff
| over, to check that you're doing that for the right reasons,
| rather than expediting shit.
| bobmaxup wrote:
| ...said no one who inherited the maintenance of hundreds of
| thousands of lines of copy pasted code.
| osigurdson wrote:
| I'm not a younger developer but I also speak snidely about
| SOLID and DRY. I also care a lot about code quality.
| happytiger wrote:
| > _We further find that the percentage of 'added code' and
| 'copy/pasted code' is increasing in proportion to 'updated,'
| 'deleted,' and 'moved 'code. In this regard, AI-generated code
| resembles an itinerant contributor, prone to violate the DRY-ness
| [don't repeat yourself] of the repos visited._
|
| Couldn't this equally be explained by the cost of refactoring
| becoming incredibly cheap? If most of your code is generated, and
| you don't have to make the investment of hand crafting
| everything, aren't you naturally going to be regularly replacing
| vast tracts? Obviously the trend may have implications, but in
| large part aren't we just seeing the impact of code cheapening?
| Serious question.
| userbinator wrote:
| Those who can't code better than an LLM will welcome it, those
| who can will abhor it. Unfortunately it seems there are far more
| of the former than the latter, and they aren't going to get
| better either.
|
| Progress comes from human intellect, mediocrity comes from
| regurgitation.
| web3-is-a-scam wrote:
| I cancelled my subscription after 2 months because I was spending
| way too much mental effort going over all of the code vomit
| fixing all of the mistakes. And it was basically useless when
| trying to deal with anything non-trivial or anything to do with
| SQL (even when I frontloaded it with my entire schema).
|
| It was much less effort to just write everything myself because I
| actually know what I want to write and fixing my own mistakes was
| easier than fixing the bot's.
|
| I weep for the juniors that will be absolutely crushed by this
| garbage.
| teaearlgraycold wrote:
| Really? I've been doing web dev as a hobby for 20 years and
| professionally for 6 or 7 years. It's super helpful for me
| giving how much boilerplate there is to write and how much of
| the training set is for websites. Any time I try to get it to
| write non-trivial SQL or TypeScript types it fails hard. But
| it's still a nice time saver for writing tests, request
| handling, React boilerplate, etc.
| louthy wrote:
| This is the problem.
|
| As programmers we should be focusing effort on reducing
| boilerplate, so that it's never needed again. Instead, we've
| created a boilerplate generator.
| CuriouslyC wrote:
| The challenge there is that reducing boilerplate comes with
| opinionated magic, which rubs a lot of people the wrong
| way. So the creators add configs and code level config, and
| pretty soon you're back in boilerplate hell.
| teaearlgraycold wrote:
| Nope. The wrong abstraction is too risky vs. having no
| abstraction. AI can let us move towards an abstraction-free
| mode of coding. In the future you'll tell the AI "add a 60
| second cache to every source of X data" and it'll scour
| your source code for matching code and you'll review the
| diffs. C programmers would review LLVM IR diffs. Web devs
| will review whatever compilation target they're using.
|
| We use high level languages because they improve reading
| comprehension and save us time when writing. Having a
| copilot take on a major role allows us to fundamentally
| rethink programming.
| norir wrote:
| I find it unlikely that we would be asking the ai for a
| very specific caching design. If the optimists are right,
| the ai should be able to do its own analysis and
| automatically design an architecture that is superior to
| what a human would be able to do.
|
| A job spent mostly reviewing ai generated diffs sounds
| like a level of hell beyond even Dante's imagination.
| Syntaf wrote:
| As with most things in life, moderation is key.
|
| I find co-pilot primarily useful as an auto-complete tool to
| save keystrokes when writing predictable context driven code.
|
| Writing an enum class in one window? Co-pilot can use that
| context to auto complete usage in other windows. Writing a unit
| test suite? Co-pilot can scaffold your next test case for you
| with a simple tab keystroke.
|
| Especially in the case of dynamic languages, co-pilot nicely
| compliments your intellisense
| thehours wrote:
| My experience is similar. This week I created a few DBT
| models and Co-Pilot saved a *ton* of keystrokes on the YML
| portion.
|
| It needed some hand-holding in the early parts, but it was so
| satisfying to tab autocomplete entire blocks of descriptions
| and tests once it picked up context with my preferences.
| ben_w wrote:
| > I cancelled my subscription after 2 months because I was
| spending way too much mental effort going over all of the code
| vomit fixing all of the mistakes. And it was basically useless
| when trying to deal with anything non-trivial or anything to do
| with SQL (even when I frontloaded it with my entire schema).
|
| Good to know, that means I'm still economically useful.
|
| I'm using ChatGPT rather than Copilot, and I'm surprised by how
| much it can do, but even so I wouldn't call it "good code" -- I
| use it for JavaScript, because while I can (mostly) read JS
| code, I've spent the last 14 years doing iOS professionally and
| therefore don't know what's considered best practice in
| browser-land. Nevertheless, even though (usually) I get working
| code, I can also spot it producing bad choices and (what seems
| like) oddities.
|
| > I weep for the juniors that will be absolutely crushed by
| this garbage.
|
| Indeed.
|
| You avoid the two usual mistakes I see with current AI, either
| thinking it's already game over for us or that it's a nothing-
| burger.
|
| For the latter, I normally have to roll out a quote I can't
| remember well enough to google, that's something along the
| lines of "your dog is juggling, filing taxes, and baking a
| cake, and rather than be impressed it can do any of those
| things, you're complaining it drops some balls, misses some
| figures, and the cake recipe leaves a lot to be desired".
| varispeed wrote:
| > I'm surprised by how much it can do, but even so I wouldn't
| call it "good code"
|
| You can tell it the code is bad and how, a lot of the times
| it will correct it. For some bs code that you have to write
| it is a great time saver.
| ben_w wrote:
| > You can tell it the code is bad and how, a lot of the
| times it will correct it. For some bs code that you have to
| write it is a great time saver.
|
| Yes, I've even done that to see how well it works (my
| experience is that half the time it fixes the code, the
| other half it claims to but doesn't) -- but I can do that
| because I can recognise that it's bad code.
|
| I don't expect a junior to notice bad code, and therefore I
| don't expect them to ask for a fix. (I've also spent 6
| months in a recent job just fixing code written by work
| experience students).
| varispeed wrote:
| > I don't expect a junior to notice bad code
|
| That is true. I actually seen instances of juniors
| struggling with code that doesn't work and frankly
| doesn't make sense, but they claim they wrote it :)
| philwelch wrote:
| > For the latter, I normally have to roll out a quote I can't
| remember well enough to google, that's something along the
| lines of "your dog is juggling, filing taxes, and baking a
| cake, and rather than be impressed it can do any of those
| things, you're complaining it drops some balls, misses some
| figures, and the cake recipe leaves a lot to be desired".
|
| Something can be very impressive without actually being
| useful, but that still doesn't make it useful. There's no
| market for a working dog that does a bad job of baking cakes
| and filing taxes, while dogs that can retrieve game birds or
| tackle fleeing suspects are in high demand.
| godelski wrote:
| I'm certainly impressed by a man that can dig a tunnel with
| a spoon. But you're definitely right that it's not that
| useful.
|
| But I disagree that half assed work is not useful. It's
| just lower usefulness. My laundry app isn't even half
| assed. The programmers couldn't even sort the room list
| (literal random order) or cache your most recent room. It's
| still better than the BS system they had before where I had
| to load a prepaid card and that machine was locked in a
| building that isn't open on weekends or after 6pm. I'm
| still immensely frustrated, but I don't want to go back to
| the old system.
|
| Though I'll mention that my fear is that because so many
| people see LLMs as having far more utility than they offer,
| we'll get more shit like the above instead of higher
| quality stuff. Most issues are solved for me to be
| comfortable in my life, so I definitely value quality a lot
| more. Plus, reduces a lot of mental stress as I'm not
| thinking "how can the person that made this be so dumb? How
| do you learn how to program and not know what sort is?"
| andy99 wrote:
| This is the crux of most of modern AI. Nobody debates it's
| cool. Since 2014 or so there has been no shortage of
| amazing demos of computers doing stuff many thought wasn't
| possible or required human level intelligence. But there's
| not automatically a bridge from those demos to commercial
| relevance, no matter how amazing.
| godelski wrote:
| > You avoid the two usual mistakes I see with current AI,
| either thinking it's already game over for us or that it's a
| nothing-burger.
|
| This is always really surprising to me that it appears to be
| these two camps. Though what frustrates me is that if you
| suggest something in the middle people usually assume you're
| in the opposite camp than they are. It reminds me a lot of
| politics and I'm not sure why we're so resistant to nuance
| when our whole job is typically composed of nuance.
|
| Though I'll point out, I think it is natural to complain
| about your juggling dog dropping balls or making mistakes on
| your taxes. That doesn't mean you aren't impressed. I think
| this response is increasingly common considering these dogs
| are sold as if they are super-human at these tasks. That's
| quite disappointing and our satisfaction is generally
| relative to expectations, not actual utility. If you think
| something is shit and it turns out to be just okay, you're
| happy and feel like you got a bargain. If you're expecting
| something to be great but it turns out to be just okay,
| you're upset and feel cheated. But these are different than
| saying the juggling dog is a useless pile of crap and will
| never be useful. I just want to make that clear, so we avoid
| my first paragraph.
| agumonkey wrote:
| The value I found in my short trial of gpt3 was a
| bidirectional path finder.
|
| Don't have me read 20 pages of docs just to integrate into a
| browser or a framework.. cutting the legwork essentially so I
| can keep my motivation and inspiration going.
| blibble wrote:
| > and therefore don't know what's considered best practice in
| browser-land.
|
| wanting this is probably the worst possible use case for LLM
| code vomit
| l5870uoo9y wrote:
| This is the problem with using AI for generating SQL
| statements; it doesn't know the semantics of the your database
| schema. If you are still open for a solution, I recently
| deployed a solution[1] that combines AI, db schema and simple
| way train AI to know your database schema.
|
| Essentially, you "like" correct (or manually corrected)
| generations and a vectorized version is stored and used in
| future similar generations. An example could be tell which
| table or foreign key is preferred for a specific query or that
| is should wrap columns in quotes.
|
| From my preliminary tests it works well. I was able to
| consistently make it use correct tables, foreign keys and
| quotes on table/column name for case-sensitivity using only a
| couple of trainings. Will open a public API for that soon too.
|
| [1]: https://www.sqlai.ai/
| tilwidnk wrote:
| It's "a public API", not "an public API", because of the
| consonant rule.
|
| I really worry that there are people out there who will
| anxiously mangle their company's data thinking what is being
| called AI, which doesn't exist yet, will save the day.
| godelski wrote:
| I have similar sentiment and looking at how mixed takes are, I
| think it depends on what you do. I write a lot of research code
| so I think it's unsurprising that GPT isn't too good here. But
| people I see that write code more in line with the "copy and
| paste from stackoverflow" style, get huge utility out of this.
| (This isn't a dis on that style, lots of work is repetitive and
| redundant).
|
| So I changed how I use GPT (which I do through API. Much
| cheaper btw). I use it a lot like how I would use SO in the
| first place. Get outlines, understand how certain lines might
| work (noisy process here), generate generic chunks of code
| especially from modules I'm unfamiliar with. A lot of this can
| just be seen as cutting down time searching.
|
| So, the most useful one: using it as a fuzzy search to figure
| out how to Google. This one is the most common pattern. Since
| everything on Google is so SEO optimized and Google clearly
| doesn't give a shit, I can ask GPT a question, get a noisy
| response that contains useful vernacular or keywords which I
| can then use to refine a Google search and actually filter out
| a decent amount of shit. I think people might read this comment
| and think that you should just build a LLM into Google, but no,
| what's going on is more complicated and requires the symbiosis.
| GPT is dumb, doesn't have context, but is good at being a lossy
| compression system. The whole reason this works is because I'm
| intelligent and __context aware__, and importantly, critical of
| relying on GPT's accuracy[0]. Much of this can't be easily
| conveyed to GPT and isn't just a matter of token length. So
| that said, the best way to actually improve this system is
| actually for Google to just get its shit together or some other
| search engine to replace them. Google, if you're listening, the
| best way you can make Google search better with LLMs is to: 1)
| stop enabling SEO bullshit, 2) throw bard into the side and
| have the LLM talk to you to help you refine a search. Hell, you
| can use a RL agent for 1 to just look how many times I back out
| from the links you send me or look at which links I actually
| use. Going to page 2 is a strong signal that you served shit.
|
| [0] accuracy is going to highly depend on frequency of content.
| While they dedupe data for training, they don't do great
| semantic deduping (still an unsolved problem. Even in vision).
| So accuracy still depends on frequency and you can think of
| well known high frequency knowledge as having many different
| versions, or that augmentation is built in. You get lower
| augmentation rates with specific or niche expert knowledge as
| there's little baked in augmentation and your "test set" is
| much further from the distribution of training data.
| marcyb5st wrote:
| As other said I use copilot or similar for
| scaffolding/boilerplate code. But you are right, reading almost
| correct code that you are unfamiliar with is much more
| demanding than fixing stuff I got wrong to begin with.
| thepasswordis wrote:
| Oh man this is the opposite of my experience!
|
| Copilot has replaced almost all of the annoying tedious stuff,
| especially stuff like writing (simple) SQL queries.
|
| "Parse this json and put the fields into the database where
| they belong" is a fantastic use case for copilot writing SQL.
|
| (Yes I'm sure there's an ORM plugin or some middleware I could
| write, but in an MVP, or a mock-up, that's too much pre
| optimization)
| linsomniac wrote:
| On the other end of the spectrum, I'm finding ChatGPT is helping
| reduce the friction of adding unit tests, getting started with
| docstrings (I usually need to do a lot of work on them though),
| and type annotations. Plus I've had some luck with handing code
| to ChatGPT and asking it to suggest clearer ways of writing it.
|
| For example, a few months ago I was rewriting some particularly
| tricky, opaque code from Ansible having to do with the conversion
| of symbolic "chmod" strings into numeric. Because I was coming
| from code that was so hard to reason about (a bug in it took me a
| couple days to understand and fix), I wanted something really
| obvious and well tested. ChatGPT helped with that.
| https://github.com/linsomniac/symbolicmode/blob/main/src/sym...
| pylua wrote:
| I think unit tests are suppose to induce friction to a certain
| extent. Using ai to do it seems like it is just checking a box.
| adonese wrote:
| Have been using chatgpt/ copilot for a while now and I feel I
| know better the limitations and what it can achieve most: - unit
| testing and docs, readme - when it can potentially hallucinate
|
| It helps me do things that I'd usually procrastinate from doing,
| yet I know how I can get them done. It is really a booster to
| ones performance if you know exactly what you want
| phendrenad2 wrote:
| Github Copilot is really good at writing spaghetti code. It isn't
| good at refactoring and modularizing your code. So if you're used
| to writing spaghetti code yourself, you can work with it and
| manage the complexity. If you're used to writing highly-factored
| code, the code it generates is going to require a lot of work.
| polyterative wrote:
| I convinced my boss to pay for it I love it
| havaloc wrote:
| Using GPT-4 has significantly enhanced the efficiency of my work.
| My focus is primarily on developing straightforward PHP CRUD
| applications for addressing problems in my day-to-day job. These
| applications are simple, and don't use frameworks and MVC
| structures, which makes the code generated by GPT-4, based on my
| precise instructions, easy to comprehend and usually functional
| right out of the prompt. I find if I listen to the users needs I
| can make something that addresses a pain point easily.
|
| I often request modifications to code segments, typically around
| 25 lines, to alter the reporting features to meet specific needs,
| such as group X and total Y on this page. GPT-4 responds
| accurately to these requests. After conducting a quick QA and
| test, the task is complete. This approach has been
| transformative, particularly effective for low-complexity tasks
| and clear-cut directives.
|
| This process reminds me of how a senior programmer might
| delegate: breaking down tasks into fundamental components for a
| junior coder to execute. In my case, GPT-4 acts as the junior
| programmer, providing valuable assistance at a modest cost of $20
| per month. I happily pay that out of pocket to save myself time.
|
| However, much like how a younger version of myself asked why we
| had to learn math if the calculator does it for us, I know
| understand why we do that. I think the same thing applies here.
| If you don't know the fundamentals, you won't be effective. If
| GPT-4 had been around when I learned to write PHP (don't @ me!),
| I probably wouldn't understand the fundamentals as well as I do.
| I have the benefit of learning how to do it before it was a
| thing, and then benefitting from the new tool being available.
|
| I also don't find the code quality to be any less, if anything
| what it spits out is a bit more polished (sometimes!).
| hackernewds wrote:
| precisely. we are very lucky to be during the timeline where
| ChatGPT was released during our later years, that we didn't
| have to compete with auto created code during our learning
| formative years.
| tippytippytango wrote:
| The valuable thing to a business is not code, it's the people in
| your business that understand the code.
|
| LLMs are ultimately going to be more valuable for reading code
| than writing it.
| whatever1 wrote:
| The use of generative models will cause their own demise. At the
| end they will just produce pure noise.
| nestorD wrote:
| I can definitely relate to that.
|
| I found that using ChatGPT to help me code led me to get working,
| well documented, code done much faster but it tend to also be
| somewhat more "naive" code: going for the obvious solutions even
| if their complexity is not optimal.
___________________________________________________________________
(page generated 2024-01-28 23:00 UTC)