[HN Gopher] Measuring GitHub Copilot's impact on productivity
___________________________________________________________________
Measuring GitHub Copilot's impact on productivity
Author : explosion-s
Score : 51 points
Date : 2024-03-01 18:26 UTC (4 hours ago)
(HTM) web link (cacm.acm.org)
(TXT) w3m dump (cacm.acm.org)
| hnthrowaway0328 wrote:
| ChatGPT is instrumental for me to maintain pyspark code
| efficiently. I don't want to learn it as I'm not particularly
| interested in the projects that use it. It's a lot easier to just
| learn on the fly and double check.
| MattGaiser wrote:
| So far this has been the key use case for me. Have it do the
| work I do rarely and it is not worth it for me to really learn
| in depth.
| cdme wrote:
| It's better than traditional autocomplete, but not
| transformative. ChatGPT is particularly bad with JavaScript --
| you're better off writing that yourself.
| cloudking wrote:
| Are you using GPT 3.5 or 4? Huge difference in quality with 4.
| dawnerd wrote:
| Interesting, I've found it's JS support pretty great. Wonder
| how much the codebase has a factor and how well things are
| typed/documented.
| k__ wrote:
| Thing is, Copilot doesn't seem to take the code of the actual
| installed packets into account.
|
| The moment you work with something a bit more obscure, it
| just guesses.
|
| Especially with TypeScript it's kinda funny, since
| IntelliSense which is right there, in the IDE, tells the
| truth, but is completely ignored by Copilot.
| k__ wrote:
| Yes.
|
| It feels like how auto complete should actually work, but I
| haven't seen any exceptional performance from Copilot.
|
| Like with Phind, the longer the code examples, the more errors.
|
| But it feels a more fluid when typing boilerplate heavy code,
| which is nice.
| nomilk wrote:
| I had an interesting experience with CoPilot yesterday. I asked
| it to add a 'deactivate' button to each of a collection of items.
| It did that, but unexpectedly, it conditionally displayed
| 'reactivate' if the `item.deactivated_at.present?` with
| appropriate bootstrap icons (a cross for deactivate and a
| circular arrow for reactivate).
|
| What surprised me was it knew what I wanted better than I did
| (for this MVP, I hadn't even considered 'reactivate'
| functionality - I'd have been happy just with deactivate).
|
| So it didn't just write code I asked for, but suggested (and was
| right about) something it thought I might like beyond what was
| actually asked for.
|
| It was my first time using CoPilot chat so perhaps it does this a
| lot, but it was my first time experiencing it. It challenged my
| thinking and improved it.
| Guthur wrote:
| How exactly had it improved your thinking? By your description
| it did it for you, you just sat back.
|
| Doing nothing does not drive self improvement, I can't get
| stronger sitting on the couch watching videos on YouTube about
| weight lifting, even if I did type in the search term.
| huytersd wrote:
| Who cares about developing skills machines can do better. I
| don't spend hours with a handsaw so I can cut perfect
| perpendicular cuts in wood, I just use a miter saw that does
| it perfectly 99% of the time. That lets me concentrate on the
| next thing on the value chain that machines can't do yet.
|
| There's plenty of woodworkers that work exclusively with hand
| tools for the joy of it and that where a lot of programming
| will be a decade in the future.
| Guthur wrote:
| That's fair, so it isn't self development just like i said?
| glad you agree.
| konschubert wrote:
| It challenges and improves your thinking just like a coding
| buddy improves and challenges you.
|
| It's not like watching others lift, it's like having a
| sparring partner. ( A very fast, slightly stupid sparring
| partner...)
|
| Coding a button isn't the part of software development that's
| intellectually stimulating, so I am glad we have robots to
| help us speed it up.
|
| I feel like you're coming from a place of disliking chatbots
| and then working backwards.
| mashlol wrote:
| Because they learned for next time?
|
| Are you suggesting that you wouldn't want anyone (or
| anything) to suggest improvements for you because then you
| would not learn yourself how to improve?
| Guthur wrote:
| Learning is hard, what was described didn't sound hard to
| me?
|
| You can call productive if you want, but sounds pretty far
| from self development to me.
| etse wrote:
| By "It challenged my thinking and improved it." OP probably
| meant 'improved upon it', or 'improved on it' where "it" was
| the originally desired output.
| BerislavLopac wrote:
| It didn't "know what you wanted" - it simply did what has been
| done many times in the code it was trained on.
| badrequest wrote:
| This feels like a distinction without a difference.
| notpachet wrote:
| Depends entirely on how close your desires are relative to
| the center of the bell curve.
| elliottkember wrote:
| Learning from experience is learning!
| shmoogy wrote:
| I find things like this too. I add detailed comments and get
| thoughtful autocompletes because many day to day things are
| solved problems.
| holoduke wrote:
| I have the same experiences, but i am using already for a year.
| Quite surprised now and then how its able to predict my next
| move. Sometimes remarkably accurate. I noticed here on
| hackernews there is a large group of copilot haters. I normally
| find this forum a place with the best open minded people out
| there. But not with copilot topics :)
| hn_throwaway_99 wrote:
| > While suggestion correctness is important, the driving factor
| for these improvements appears to be not correctness as such, but
| whether the suggestions are useful as a starting point for
| further development.
|
| I admit I didn't read the whole article, but that bullet I
| thought was key. I totally agree. When folks yell "BuT
| HalliciNations!!!", I get it, but that doesn't mean that LLMs can
| still be a huge boon if you know how to use them and don't just
| trust their output blindly and yolo it into production.
| scubbo wrote:
| Precisely. I'm all in favour of AI/LLM/GAN/Whatevers being used
| as _an_ input to the process of human creativity or judgement.
| They only become troublesome when their output is blindly
| trusted.
| dmix wrote:
| I love Copilot, I find it essential these days, but if it's going
| to significantly impact my productivity it's got to be 2x faster.
| The autocompete is scary good at predicting what was in my head,
| sometimes before I figure it out, but often it trails behind my
| own speed.
|
| But anyway, the future is exciting.
| konschubert wrote:
| You must either have god-like typing speed or a really slow
| Internet connection ;)
| daemonologist wrote:
| There was a Show HN a few days ago [0] from a service promising
| ~3x faster copilot-style autocomplete. I can confirm that it is
| extremely fast (sometimes too fast - I use ctrl+right arrow a
| lot to navigate, which with most completion services isn't a
| problem because I'll have moved the cursor before a completion
| comes back, but with supermaven I've been accidentally
| accepting the completion instead).
|
| [0] https://news.ycombinator.com/item?id=39473773
| mpweiher wrote:
| The outcome they are measuring is "perceived productivity", which
| seems pretty weak sauce to me.
|
| _Here, we investigate whether usage measurements of developer
| interactions with GitHub Copilot can predict perceived
| productivity as reported by developers._
|
| And they use the "actual activity" to predict this perceived
| productivity.
| bottlepalm wrote:
| I'm already an experienced developer, but writing code in new
| domains is sooo much nicer now. I can learn and get things done
| way faster than ever before. Copilot has pretty much replaced
| Google and Stack Overflow for me. I use it all day everyday. The
| chat feature is great at well to discuss code, questions, ideas,
| etc.. I still use ChatGPT 4 for bigger questions, more complex
| things, writing entire files, etc..
| dotnet00 wrote:
| Copilot has been more impactful for 'unproductive' stuff for me.
| Not as good at handling the code related to my job, as it's a
| math heavy beast in an extremely niche field. But great for my
| hobby, writing little bots to toy around with various things.
| Lets me skip remembering or looking up a lot of basic
| implementation details of talking to the service APIs,
| drastically speeding up the time to get a functional prototype.
|
| As others have mentioned, it's great for speeding up all the
| little boilerplate and other things too simple or otherwise too
| unrelated to the main goal.
|
| Sometimes it almost works like rubberduck-development, by seeing
| what Copilot spits out, I have sometimes realized earlier than
| otherwise, that I missed certain components/checks that I should
| probably plan ahead for (eg remembering to add a convenient way
| to handle user profiles for a bot).
| yodon wrote:
| For me, Figure 6 is the most interesting observation: programmers
| are interacting with Copilot is a statistically different way
| nights and weekends vs working hours.
|
| I'm guessing (without hard evidence) that this implies day job
| code reviewed commits and weekend hobby project/side
| hustle/startup coding are held to different standards by the
| developers involved.
| bisby wrote:
| Or if doing your day job during off hours, you're just looking
| for a quick solution and don't actually care about polish
| because you just want to get back to not working.
| sagman wrote:
| I use it for doing mundane tasks like creating queries following
| a pattern I defined or creating docs for code. My experience has
| been great so far. Not sure if it is worth the price for my
| company, but it encourages me being a little lazy and saves some
| time.
| munk-a wrote:
| I think it's too early to tell but my main concern about copilot
| is code maintainability and security. Copilot is able to barf out
| helpful expressions that will reduce the amount of code we need
| to write by hand - I think it's excellent when it comes to
| reducing boilerplate... but I think a large amount of boilerplate
| existing belies a bigger issue with the project. The majority of
| software engineering isn't writing code - copilot may be
| beneficial as an accessibility aide for developers that have
| typing impairments but most developers can type faster than they
| can think - if the level of boilerplate in your project is
| reasonable then this should mean you're never prevented from
| thinking because your fingers are still working on recording your
| previous thought. However, at the end of the day, if you can help
| reduce carpal tunnel that's still a win.
|
| The problem I can foresee with copilot is that the scenario
| change you're agreeing to is that you'll type less but need to
| read over the code produced more - this is an effort that isn't
| normally necessary (typos happen but those should take a trivial
| time to correct) but when copilot is involved you need to proof
| all the code that is being generated. There is a motivation to
| skip this step and just accept the code was written correctly and
| that will inevitably lead to security problems - and there is a
| motivation to not correct or alter auto-filled command. If
| there's a multi-dimensional array and you think it semantically
| makes sense to iterate it over dimension a then dimension b and
| copilot instead goes with b as the major index then it's more
| likely to remain in a b major iteration - that may make code less
| readable or it may cause major issues down the line.
|
| Copilot, IMO, is optimizing the least important part of
| development right now and it costs us more to correct it then it
| would to just splat out the correct code _but_ this is a similar
| argument to longbows vs. crossbows - hand a peasant a crossbow
| and they can fire a crossbow - train a peasant for 30 years and
| they can fire a longbow - the longbow is more powerful, but the
| crossbow is a clear choice in terms of RoI. It may be that
| today's developers will only benefit from copilot minimally since
| we've invested the training time in standard development
| practices but tomorrow's developers will eschew a lot of the
| algorithmic learning and still be able to deliver the majority of
| the value.
| Zambyte wrote:
| > I think it's excellent when it comes to reducing boilerplate
|
| I think the rest of your comment already elaborates this point,
| but I want to highlight that language models do not reduce
| boilerplate, they automate it. Reducing boilerplate happens by
| designing simpler APIs, not by making language models interface
| with complex APIs for you.
| WirelessGigabit wrote:
| What I am most looking forward to is future changes in
| development speed. I wish the report tracked the changes over
| time. Were they committed as-is (which is a metric to the local
| quality of the suggestion) and how long does it survive (which is
| a metric to the global quality of the code).
| richardw wrote:
| [delayed]
___________________________________________________________________
(page generated 2024-03-01 23:00 UTC)