[HN Gopher] Benchmarking the accuracy of GPT3.5's and GPT-4's co...
___________________________________________________________________
Benchmarking the accuracy of GPT3.5's and GPT-4's code generation
abilities
Author : seanmor5
Score : 60 points
Date : 2023-04-06 20:08 UTC (2 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| dontupvoteme wrote:
| I have found somewhat interesting results by translating my
| prompts into other languages (using deepl) -- I haven't run the
| statistics in depth but German and French results tend to have
| more comments. Japanese uses variable names i, j, etc. I suspect
| languages which use e.g. cyrillic will produce significantly
| different results - but the tokenizer also "punishes" them in the
| sense that they're significantly more expensive
|
| One area of low hanging fruit here is to automatically evaluate
| the quality/accuracy/correctness/etc of a given generation and
| select (or merge) between multiple possibilities generated in
| parallel. Sometimes it will forget to fill in a function def, so
| use the one from iter #3, etc. You could go so far as to run
| candidates in a sandbox with an input and evaluate which one
| gives output -- ideally which output is closest to what is
| desired, if you can define that.
|
| Also a sort of "whitelist" for valid functions and routines -
| sometimes it's close not still wrong, if you can map the
| hallucinations and mistakes to what it's supposed to be, that
| also can probably go a long way.
| youssefabdelm wrote:
| TL;DR?
|
| (Yes, GPT-4 is prob better... but by how much and on what?) A
| table would've been easier
| HopenHeyHi wrote:
| You just couldn't scroll to the bottom of the page, eh?
|
| The conclusion is that neither 3.5 nor 4 are good enough
| because for anything none trivial they generate code that is
| often subtly wrong. Might still speed up somebody new to the
| language/project/learning or I would say: with additional
| tooling/plugins/"prompt engineering"/tinkering the author might
| get useful results.
| hammyhavoc wrote:
| Not the creator, but as it's GitHub, feel free to repeat the
| experiment and submit a pull request with a better method; peer
| review.
| beebmam wrote:
| Not everyone has the time or expertise to create a pull
| request. But there are issues allowed on this repo! Create an
| Issue if you'd like the author to address something, in my
| opinion.
| fullsend wrote:
| I've seen a good number of things get addressed simply
| because an Issue crossed a threshold of votes/comments over
| time. Really can make voices heard on the anon internet.
| Big respect to anyone working for free who takes them
| seriously, that takes integrity.
| dwohnitmok wrote:
| > So, should you use GPT to generate your OpenAPI validations?
| Probably not... yet... I'm looking forward to repeating this
| experiment with GPT-6, and maybe GPT-7 will be able to generate
| an JSONSchema compiler and replace this library altogether.
|
| from
| https://github.com/E-xyza/Exonerate/blob/master/bench/report...
|
| (I believe the author is significantly underestimating the pace
| of progress)
|
| Specific numbers are at https://github.com/E-xyza/Exonerate/blo
| b/master/bench/report.... GPT-4 does significantly better.
| iForgotMyPW wrote:
| [dead]
| dwohnitmok wrote:
| The conclusions start from
| https://github.com/E-xyza/Exonerate/blob/master/bench/report...
|
| This is particularly impressive for Elixir, which is not a
| language that is a particular focus of GPT-4. I imagine the
| accuracy for Python is extremely good. Maybe near perfect for
| this kind of benchmark if allowed to see error messages.
| naet wrote:
| It's also possible GPT-4 is better at writing Elixir since
| there are less beginners/students writing Elixir code and
| polluting the training data with bad practice or faulty code.
| coder-3 wrote:
| This might not be an issue for the same (somewhat
| inscrutable) reason that GPT-4 has quasi-perfect grammar.
| JackFr wrote:
| Not to split hairs, but John Henry competed against a steam
| drill. I'm not sure what an iron track layer is...
| nomel wrote:
| I don't see any iterations. I can sometimes get it to improve the
| code just by telling it that there's a problem, or to identify
| any problems.
|
| > it's not clear if GPT will have sufficient attention to handle
| the more complex cases.
|
| I think this could be helped with some tooling, by giving it
| smaller pieces of code to digest, in a new prompt and a somewhat
| smaller prompt. These LLM can't seem to dive into cracks if the
| context isn't constrained to that crack.
| throwawaymaths wrote:
| > just by telling it that there's a problem
|
| how will you identify that there's a problem?
| blibble wrote:
| especially as a new developer that now thinks they will never
| have to write any code
| sosodev wrote:
| Well in a lot of cases the model is writing code that can't
| even be run (the red dots). Feeding it the compilation error
| can be done automatically and it will usually be able to at
| least get it running.
| nomel wrote:
| To answer your question from a practical perspective, you can
| try to run it, and feed back errors. See:
| https://news.ycombinator.com/item?id=35446171
|
| But that's actually not what I meant. You can often just tell
| it "there's a problem, please fix it", or "do you see any
| problems" and it will be able to identify it without
| additional input. There no requirement that you've identified
| a problem, it's more of a "double check that" type of prompt.
| bastardoperator wrote:
| Do you have access to the code interpreter alpha? You can
| upload larger chunks of code and have it perform different
| tasks. One of the interesting features is that chatgpt will
| sometimes have some issues with its own code (python) and try
| to automatically correct itself.
| WXLCKNO wrote:
| I wish I had access to the plugins, both as a user or
| developer. I hate the feeling of being on the outside of the
| latest developments, nothing feels open about a OpenAI.
___________________________________________________________________
(page generated 2023-04-06 23:01 UTC)