[HN Gopher] Can programming languages boost each other via instr...
___________________________________________________________________
Can programming languages boost each other via instruction tuning?
Author : og_kalu
Score : 70 points
Date : 2023-09-01 15:03 UTC (7 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| wcoenen wrote:
| It was already known that training on code improves performance
| on all reasoning tasks[1], so this is not really surprising.
|
| [1] https://arxiv.org/abs/2210.07128
| [deleted]
| air7 wrote:
| Wow, this is the first contradiction to Betteridge's law that
| I've encountered...
|
| https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
| pmontra wrote:
| Betteridge's Law can be trivially confuted. For example one
| could make a blog post titled "Did the Sun Raise Yesterday?" I
| don't dare trying the law with a headline like that about
| tomorrow though.
| lainga wrote:
| Sort of. None of the programming languages mentioned in the
| paper were actually boosted in any way, an LLM was.
| cmdrk wrote:
| is it the programming languages themselves, or the programmer
| comments within the training sets that improve output?
| sp332 wrote:
| There are comments in the data, but 95% of the comments in the
| HTML one seem to be variations of /*CSS styles go here*/.
| pipo234 wrote:
| > More surprisingly, we found that CodeM-HTML 7B trained on the
| HTML corpus can improve Java by an absolute 15.24% pass@1.
|
| (Tongue in cheek reaction:)
|
| Any sufficiently advanced technology is indistinguishable from
| magic -- Arthur C. Clarke
| xeckr wrote:
| Compared to 17% when pre-trained on Python. Looks like the
| debate on whether or not HTML is a programming language is
| settled!
| codetrotter wrote:
| Was the HTML corpus it trained on only HTML and text, or was
| there JS code in the HTML?
| sp332 wrote:
| There sure was. Here is a snippet from the very first
| example in the html.json file: <input
| type="text" id="searchQuery" name="searchQuery"
| placeholder="Enter name..."> <button
| onclick="searchEmployees()">Search</button> </div>
| <script> function searchEmployees() {
| var input, filter, table, tr, td, i, txtValue;
| ...etc...
|
| There's no code in the repo, but the paper says this about
| how the inputs were generated:
|
| _For all the selected languages except HTML, we adopt an
| in-depth evolution, by asking GPT-3.5 to rewrite the seed
| instruction (Python) into a more complicated version
| relevant to the target language (Python, JavaScript,
| TypeScript, C, C++, Java, or Go). However, for HTML, we
| adopt in-breadth evolution to produce a brand-new HTML-
| related instruction..._
| AtlasBarfed wrote:
| I've just realized that the most ultimate "no good deed goes
| unpunished" is that the mountain of open source code released by
| programmers will be used by LLMs to put them out of jobs.
| csjh wrote:
| Fortunately we'll get revolutionary new products from it, such
| as calendar app (with AI integration), email app (with AI
| integration), resume builder app (with AI integration), and
| more!
| sumtechguy wrote:
| oh no and a wave of patents with 'using ai' it will be 1999
| all over again.
| munificent wrote:
| And the mountain of concept art created by digital artists and
| uploaded to the web is putting them out of jobs thanks to
| DALL*E and friends.
|
| And the mountain of writing created by authors and uploaded to
| the web is putting them out of jobs.
|
| And the mountain of music created by musicians is going to
| start putting them out of jobs.
|
| Generative AI is an unprecedented transfer of wealth-generating
| capability from creative people to giant corporations.
| blibble wrote:
| in a world where all office workers are now permanently
| unemployable: what use are Microsoft's products?
|
| AI kills 90% of demand for their products
|
| I don't think roofers need Microsoft Teams
| bitwize wrote:
| But Microsoft doesn't sell to office workers, they sell to
| executives... who would relish the idea of getting rid of
| those office workers. And guess who owns "Open" "AI"...
| blibble wrote:
| will the executives buy legions of Windows machines if
| they don't have any workers?
| bcrosby95 wrote:
| And it will occur faster than any transition in history.
| Meatspace has natural limits to how fast things can do this.
| staunton wrote:
| But, contrary to at any such developments in history, the
| people who are losing out aren't the "poor working class".
| They already have cultural and political power that they
| stand to lose. This might still go a lot of ways.
| Timon3 wrote:
| Some of them sure, but you're painting with the broadest
| brush possible. Do you really think the average artist is
| not "poor working class", that they have cultural and
| political power?
| staunton wrote:
| Maybe not the average but the power is concentrated in a
| few and they are also affected. I'm expecting many who
| collectively do have power to participate in shaping how
| these things are used and perhaps oppose the big
| companies.
| [deleted]
| netbioserror wrote:
| So the equivalent of transfer learning. Neat, but I'd like to see
| it forced into a new paradigm or syntax. How would it handle
| Haskell or Clojure?
| SomewhatLikely wrote:
| They do show training on HTML is still leading to improvements
| across languages. It looks to me like training on C led to the
| biggest average improvement among the tested languages. I'm
| curious if the effect translates to humans as well. I know that
| for myself learning C taught me a lot about what the higher
| languages are abstracting away.
| Filligree wrote:
| It's a common recommendation, right? I think this might be
| close to being actual evidence for it.
| ahoka wrote:
| The risk for permanent brain damage is too high. Although
| knowing C helps understanding why some horrible design
| elements exist in "modern" programming languages.
| catskul2 wrote:
| Since this is on arxiv one presumes it's not been peer reviewed
| yet, right? Is there a way to leave comments? There are a couple
| of confusing language/grammar mistakes including some in the
| first sentence.
| behnamoh wrote:
| If there's anything I've learnt in the past year about LLMs,
| it's this:
|
| there's so much hype, ungrounded research "findings", and next-
| big-thing libraries (remember langchain?)
|
| LLMs, at least the ones companies use, are APIs all the way
| down. Stay away from anything or anyone who claims they've
| discovered something no one else knew about LLMs.
| aalimov_ wrote:
| Maybe staying away from someone selling something while
| making that claim, but staying curious seems appropriate.
|
| > LLMs, at least the ones companies use, are APIs all the way
| down.
|
| Really does seem like it. So many interesting little projects
| that end up being nothing more than a hidden prompt and an
| api call to OpenAI
| mcphage wrote:
| Even the first sentence of the summary is unclear:
|
| > When human programmers have mastered a programming language,
| it would be easier when they learn a new programming language.
|
| I think they're saying "When human programmers have mastered a
| programming language, it makes it easier for them to learn a
| new programming language", but... I'm not entirely sure.
| toast0 wrote:
| Yeah, that's what they're saying. English is not an easy
| language.
|
| "It would be easier to learn a programming language, when you
| have mastered any other programming language" is clear, IMHO,
| but flipping the clauses makes it less clear.
| HappMacDonald wrote:
| > English is not an easy language.
|
| "ChatGPT, please translate my incoherent stream of
| consciousness into clear English"
|
| then review its output like one would a rough draft, and
| repeat as needed.
|
| As far as human languages go, yes English is at least
| B-tier difficulty. Not least of which due to having the
| largest working vocabulary on Earth. But for all of the
| problems that LLMs might have with object permanence and
| memory and such, I think it's pretty safe to say that GPT3+
| has _mastered_ both the commonly used vocabulary and even
| the most subtle turns of grammar in the English Language.
| Which is never going to not dumbfound me. O_O
| scott_s wrote:
| Email the first author.
| leetharris wrote:
| This is true with natural language as well in LLMs, ASRs, and
| more.
|
| At Rev we found that adding data from other languages to
| primarily-English ASR models helped reduce WER in the target
| language substantially (especially when accents were involved).
| We found similar benefits when training translation models.
___________________________________________________________________
(page generated 2023-09-01 23:01 UTC)