[HN Gopher] Art or Artifice? Large Language Models and the False...
___________________________________________________________________
Art or Artifice? Large Language Models and the False Promise of
Creativity
Author : p4bl0
Score : 49 points
Date : 2023-10-02 19:53 UTC (3 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| benreesman wrote:
| Yes the models got a straightforward prompt, and they didn't get
| re-rolled until it was good, and the comparison was to real
| writers.
|
| The conclusion isn't that the models are useless or something:
| it's that they aren't a replacement for people.
|
| This a cake _or_ eat it moment: AI is a powerful tool that in the
| hands of those who embrace it _or_ it's trivially superhuman in a
| broad-spectrum way.
| kromem wrote:
| The prompt used: "Write a New Yorker-style story given the plot
| below. Make sure it is at least {{word_count}} words. Directly
| start with the story, do not say things like 'Here's the story
| [...]"
|
| They didn't even bother using multiple passes to prompt better
| quality creative writing like "Write an outline for the story..."
| "Expand this outline focusing on improving style and tone..."
| "Here's a story written by an amateur writer. As a professional
| writer, give it a rewrite improving vocabulary, metaphor, and
| overall quality."
|
| As with a lot of the LLM stuff I see these days, I have to wonder
| at how much of what's being measured is the capacity of the tool
| and how much the measurement of the capacity of its users.
|
| I'd never imagine using a LLM zero shot in a single pass for any
| creative writing tasks, and measuring the inability to perform in
| suboptimal conditions isn't all that revealing or novel (pun
| intended).
| hanselot wrote:
| Or my favourite thing to add as custom instruction: Rewrite
| this amateur's code like the lovechild of Molenaar (RIP) and
| van Rossum.
| joe_the_user wrote:
| If you are testing a system, you can't simultaneously _tune_
| the system - or at least, if you 're establishing a new testing
| approach you can't once do. Maybe the testing approach is
| established, someone could experiment with ways to improve the
| system on the benchmark. But if you're establishing a
| benchmark, improving at the same time is going to make such a
| benchmark kind of meaningless.
| bloopernova wrote:
| Is it a good idea to do multiple passes when asking for code?
|
| What sort of progression do we see in a multipass (!) workflow?
| Does anyone have any examples?
|
| (Wrangling AIs is a skill, I think, and thus learning more is
| always a good idea)
| causality0 wrote:
| An interesting study but a horrible title. Calling the entire
| concept of creativity by LLMs a "false promise" in 2023, three
| years after the launch of the first commercial LLMs, like someone
| in 1897 three years after the first production car calling the
| automobile a "false promise of commuting".
| vannevar wrote:
| Right---it's like comparing ten-year-old Hemingway's writing to
| 40-year-old Hemingway's writing.
| kenjackson wrote:
| They're comparing LLM output to professional creative writers?
| That seems like a high bar to start with. I'd think that would be
| the highest bar they could achieve. How does it compare though to
| your average college student?
|
| Also I feel like I haven't heard people say that much about their
| ability to be creative. It's more about their ability to generate
| human like content -- and most human content isn't that creative
| either.
| ToucanLoucan wrote:
| > They're comparing LLM output to professional creative
| writers?
|
| I mean that seems a fair experiment to judge based on the
| rhetoric employed by AI enthusiasts, who talk of things like
| "democratizing art" (as though art as it exists is somehow
| anti-democratic?) and letting any layman slam some text into a
| prompt and get a work of shakespeare, a painting by Da Vinci,
| or a Mozart symphony back.
|
| I think it's nonsense but they don't, so test it?
| vannevar wrote:
| Yes, it's a testament to how powerful these models have become
| in a such a short time that we've gone from being impressed
| that they can write a coherent paragraph to being critical that
| they can't win a professional writing competition, all in the
| span of 5 years.
| loa_in_ wrote:
| We is a broad term. Do we really count surfacing of an
| article like that as we being critical?
| vannevar wrote:
| More accurate to say "some" being critical, sure. But we're
| talking about it, so I think the juxtaposition is fair.
| krapp wrote:
| I mean, professional writers are already being replaced by LLMs
| so it's a fair bar, if high.
| og_kalu wrote:
| Yeah GPT-4, the current Sota isn't generally
| professional/expert level in most domains so this isn't exactly
| surprising revelation even if true.
|
| This paper also suffers from the "single pass problem". LLMs
| aren't that new anymore. Most researchers should know that
| whatever you get on a single pass can be/is often much lower
| quality than if taking the effort to be more specific, provide
| examples or breaking the task down and going through multiple
| passes.
|
| From my experience getting several LLMs to write, this doesn't
| hold any less true in the domain of creative writing so there's
| a very good chance this paper severely underestimates the
| potential of even current LLMs.
| commandlinefan wrote:
| I'm seeing a lot of AI-generated images on my facebook feed
| (they're of, um, predictable subjects) - if a human generated
| one, it would definitely be considered "art". Is Shrek not art
| because computers generated it based on input from humans?
| krapp wrote:
| > Is Shrek not art because computers generated it based on
| input from humans?
|
| Computers didn't "generate" Shrek based on input from humans -
| human artists used their creativity and expertise to design
| those models, build the environments and animate the scenes.
| Those people went to college for years and studied and honed
| their craft, and applied it.
|
| If you can't see the difference between that and typing words
| into a prompt and having a machine stochastically generate
| something that hopefully kind of resembles what you want, but
| that you wouldn't be able to create on your own, and that you
| have no real ability to edit or fine tune, then I don't know
| what to tell you.
|
| And I agree with you - a lot of what AI is generating now looks
| really good. The thing is, even if it could be considered art,
| no one using those tools should consider themselves an artist.
| They didn't create anything, they're not capable of creating
| anything, and as long as they keep using AI, they will never be
| capable of creating anything. They're cheating themselves out
| of the experience of real creativity for instant gratification
| and a hit of endorphines.
|
| Which is fine as long as that's accepted. That's got to be the
| deal with AI - you give up the right to ever be talented at
| anything for the instant gratification machine that makes
| generic pretty pictures, and you'll never be anything more than
| a masturbating monkey until you put it aside and actually learn
| something.
| iamacyborg wrote:
| A lot of generic images aren't art, regardless of how they're
| produced. Much the same way as someone's holiday snapshots are
| not art, even if some photography definitely is.
| bluetwo wrote:
| I appreciate the attempt to create objective evaluations by
| leveraging existing tools. It isn't a perfect experiment or
| article, but I like the goal.
| caesil wrote:
| >We prompt three top-performing LLMs: GPT3.5, GPT4, and Claude
| V1.3 to generate a story of similar length to each New Yorker
| story, based on the one-sentence plot summary
|
| This is kinda like resizing an image down to 100x100px and then
| asking AI to upscale it and comparing against the original. Of
| course the attempts to reverse lossy compression won't be as
| good, even if the system is capable of similar quality work when
| correctly prompted.
| vannevar wrote:
| Not really. The point of the LLM exercise is to measure
| creativity. Upscaling is not a creative process, it's basically
| the opposite of creativity. The only reason to keep the prompt
| in the same vein as the original story is for the judges, to at
| least keep the premise in the same ballpark so they're
| comparing comparable stories.
| letmevoteplease wrote:
| The most interesting part to me is that Claude 1.3 outperformed
| GPT-4.[1] They also note that Claude stories are "more likely to
| be attributed to an amateur writer than an AI, whereas GPT3.5 and
| GPT4 stories are 80%+ attributed to AI."
|
| [1] https://i.imgur.com/TnOw8Jw.jpg
| og_kalu wrote:
| That's honestly not surprising. Whatever RLHF Open ai does to
| the GPT models really messes with its creative writing. Like a
| hallmark filter over output if you don't try to steer it away.
| The Instruct models don't have the same problem to that degree.
|
| I really hope Open ai releases a GPT-4 Instruct model, for this
| reason and for chess.
___________________________________________________________________
(page generated 2023-10-02 23:01 UTC)