[HN Gopher] A Minimalist Guide To Program Synthesis
___________________________________________________________________
A Minimalist Guide To Program Synthesis
Author : optimalsolver
Score : 98 points
Date : 2022-10-14 14:29 UTC (8 hours ago)
(HTM) web link (evanthebouncy.github.io)
(TXT) w3m dump (evanthebouncy.github.io)
| zomglings wrote:
| Thank you for posting this. Code generation is an interest of
| mine and a core foundation of the technology that I build.
|
| It will take me some time to read through this blog, but I have a
| question:
|
| Is there any research being done where people are using large
| language models to generate or transform syntax trees for
| programs as opposed to operating with programs as simply streams
| of tokens?
| le-mark wrote:
| > large language models
|
| Can you clarify what this means? In this context seems like it
| could mean either machine learning language model or
| programming language grammar/semantics.
| evanthebouncy wrote:
| I meant gpt3 and its cousins.
| uoaei wrote:
| LLM is a colloquial term for the latest greatest NLP text
| generation / AI chatbot models.
| zomglings wrote:
| Not sure if there is a technical definition of the term. I am
| using it to refer to the transformer-based neural networks
| that are used to build models like Open AI's Codex (which
| GitHub Copilot uses IIRC).
|
| I believe they are called large because they contain orders
| of magnitude more parameters than the models that came before
| them. They do not fit in memory on a single training device
| (GPU or TPU usually) and require serious infrastructure work
| to train effectively.
|
| The models are also trained on large datasets. These datasets
| are usually text streams. In the case of GitHub Copilot, it
| was the textual source code of a large percentage of the git
| repositories on GitHub. (Maybe supplemented with other code?)
|
| I am curious if anyone is parsing source code into syntax
| trees and then training large language models to operate on
| the resulting graph structures.
|
| Edit: On a related note - GitHub Copilot is extremely good at
| generating syntactically correct code. Has anyone studied
| which parameters in the language model are related to
| learning programming language syntax? Are the parameters in
| the model related to javascript syntax "close to" the
| parameters in the model which learn Go syntax?
|
| Edit 2: For GitHub Copilot, did they use the content of git
| diffs to learn the kinds of _edits_ people make to existing
| code?
|
| Edit 3: I realize that a random HN post about some blog is
| probably not the best place to be asking these questions. If
| anyone could point me to a better place, I would appreciate
| it.
| evanthebouncy wrote:
| Here's an obtuse answer.
|
| The operations that transform syntax trees are just programs as
| well which... Can be represented as a steams of tokens.
|
| But this is likely helpful if it makes it easier to learn by
| the NN. I'm certain there are works that do this, but not off
| the top of my head rn
| mindcrime wrote:
| _Is there any research being done where people are using large
| language models to generate or transform syntax trees for
| programs_
|
| I saw a mention of this on Twitter the other day. I have not
| read the paper, and it's anonymous, so I don't know how serious
| and/or credible this is. But it was apparently an actual ICLR
| submission, so it might be worth a look. Not sure it's exactly
| what you're interested in, but it seems like it might be in the
| same neighborhood.
|
| https://openreview.net/forum?id=SKat5ZX5RET
| zomglings wrote:
| Thanks for the link. This is interesting work, although not
| related to operations on program syntax trees.
|
| From what I understand, they are using reinforcement learning
| techniques inspired by AutoML to learn architectures for
| language models that are effective at code generation.
|
| They have reinforcement learning models optimizing the neural
| networks that generate code.
|
| Pretty cool and reminds of work that followed on the heels of
| AutoML to build neural networks to automatically solve
| arbitrary ML problems (with the problem as the input
| parameter).
| evanthebouncy wrote:
| Hiyo I'm the author xD
|
| Feel free to ask me questions uhhh... How does hacker news work
| lmao do i get notifications even hmm.
|
| I'll uhh... refresh this page once in awhile and see if I can
| help!
|
| if that don't work prolly just ping me on twitter.
| fab1an wrote:
| This is a GREAT resource, thanks for posting.
|
| I feel the advent of GPT-3 / Codex and the power that comes with
| it has been surprisingly underestimated - maybe because the folks
| who would benefit most from it (people who don't write code, like
| myself) haven't really caught up with it, while the ones using it
| (mostly people who do write code) maybe don't see the immediate
| benefits as much (and maybe are a little bit concerned about what
| this will mean to their economic position in all this...?)
|
| I've played around a ton with simple GPT-3 code over the last few
| weeks and am in total awe of the possibilities this opens up to
| folks like me (who don't write code)
|
| E.g. I had GPT-3 write me app script integrations of itself (!)
| and stablediffusino into a Google Sheet, allowing me to build a
| massive sandbox playground for various prototyping experiments.
|
| Here's a thread on that process if you're curious:
| https://twitter.com/fabianstelzer/status/1577416649514295297
| evanthebouncy wrote:
| yes. reading some of your twitter posts I'd say I share your
| sentiment on how codex will upend the programming world. but I
| think better integrations and statement of intents need to be
| more fleshed out, given openAI & friends are tech companies and
| not user companies, the UX part will take time.
| ShamelessC wrote:
| That is the opposite of my experience. Being a skilled coder
| means I can verify the outputs quickly. Further using a
| beginner coding "style" actually encourages the model to output
| similarly "beginner" code.
|
| It has been the largest boon to my productivity in years. I
| code exclusively in Python where copilot is quite good. In my
| experience the comments of supposed "experienced" coders mad
| about copilot are either GPL proponents or people who work with
| mostly statically typed languages. Almost none of them have
| actually used Copilot and certainly didn't use it long enough
| to ascertain a "good faith" criticism of it (so not just "it
| outputs subtle bugs throw it in the trash!")
|
| There's also just a natural reaction to hate things poised to
| make you redundant ( probably should have considered this when
| becoming a coder, though).
| rngname22 wrote:
| Would be really interesting to see a pluggable / integration-y
| automaton-builder company like Zapier tackle adding ML models to
| their abilities
| danielvaughn wrote:
| Kind of unrelated, but I love this blogs typography.
| evanthebouncy wrote:
| it is very related, and I spent an ungodly amount of time to
| make sure it is visually compact and to the point, so I can
| deliver knowledge directly into your eye holes with minimal
| friction. It's super heartwarming to see someone appreciating
| this aspect.
___________________________________________________________________
(page generated 2022-10-14 23:01 UTC)