[HN Gopher] GenAI Image Editing Showdown
___________________________________________________________________
GenAI Image Editing Showdown
Author : rzk
Score : 176 points
Date : 2025-10-26 02:57 UTC (20 hours ago)
(HTM) web link (genai-showdown.specr.net)
(TXT) w3m dump (genai-showdown.specr.net)
| isoprophlex wrote:
| The "editing" showdown is very good. Introduced me to the
| Seedream model which i didn't know about until now.
|
| I don't fully understand the iterative methodology tho - they
| allow multiple attempts, which are judged by another multimodal
| llm? Won't they have limited accuracy in itself?
| ACCount37 wrote:
| "LLMs judged by LLMs" is the industry standard. Can't put a
| human judge in a box and have him evaluate and rate a set of
| 7600 responses on demand.
|
| Now, are LLM judges flawed? Obviously. But they are more shelf
| stable than humans, so it's easier to compare different
| results. And as long as you use an LLM judge as a performance
| thermometer and not a direct optimization target, you aren't
| going to be facing too many issues from that.
|
| If you are using an LLM judge as a direct optimization target
| though? You'll see some funny things happen. Like GPT-5 prose.
| Which isn't even the weirdest it gets.
| vunderba wrote:
| I tried to make the judgement criteria more clear in the FAQ
| section - I'll post it here:
|
| _What is the metric on which these models are being judged?_
| It's hard to define a discrete rubric for grading at an
| inherently qualitative level. To keep things simple, this test
| is purely PASS/FAIL - unsuccessful means that the model NEVER
| managed to generate an image adhering to the prompt. For
| example, Midjourney 7 did not manage to generate the correct
| vertical stack of translucent cubes ordered by color in 64
| generation attempts. In many cases, we often attempt a generous
| intepretation of the prompt - if it gets close enough, we might
| consider it a pass. Put another way: if I were to
| show the final image to a random stranger on the street, would
| they be able to guess what the original prompt was? (aka the
| Pictionary test). To paraphrase former Supreme Court
| Justice Potter Stewart, "I may not be able to define a passing
| image, but I know it when I see it."
|
| To answer your question, the pass/fail is manually determined
| according to a set of well-defined criteria which is usually
| specified alongside the image.
| sans_souse wrote:
| I had to upvote immediately once I got to Alexander the Great on
| a Hippity Hop
| halflife wrote:
| The horse chimera is much better
| adriand wrote:
| I had completely forgotten about the hippity hop and coming
| across it here brought back all kinds of childhood memories.
| Those things were fun!
| mrec wrote:
| They were always called "space hoppers" here in the UK, and
| always looked like this:
|
| https://en.wikipedia.org/wiki/Space_hopper#/media/File:Space.
| ..
| croes wrote:
| What about the classic: A analog watch that shows the time 08:15?
|
| Did current models overcome the 10:10 bias?
| echelon wrote:
| This would be easy to patch the models to fix. Just gather a
| small amount of training data for these cases, eg. "change the
| clock hands to 5:30" with the corresponding edit.
|
| Three tuple: (original image, text edit instruction, final
| image).
|
| Easy to patch for editing models, anyway. Maybe not text to
| image models.
| fart-fart-FART wrote:
| waste of money and effort, imo. there are +- infinity of such
| small things to fix.
| echelon wrote:
| It's a common enough use case that it'll be added at some
| point.
|
| It probably comes up more than you think. Storyboarding,
| product placement, model images, etc.
|
| It's not critical in the short term, but it'll wind up on
| their backlog for sure.
| konart wrote:
| >Cephalopodic Puppet Show
|
| I'm pretty sure that only Gemini made it. Other models did not
| meet the 'each tentacle covered' criteria.
| jedbrooke wrote:
| for the OpenAI 4o model on the octopus sock puppet prompt, the
| prompt clearly states that each tentacle should have a sock
| puppet, whereas the OpenAI 4o image only has 6 puppets with 2
| tentacles being puppetless. I'm not sure if we can call that a
| pass
| snowfield wrote:
| I'd assume that behind the scenes the models generate several
| passes and only show the user the best one, that would be smart,
| as to to make it seem their model is better than others
|
| Is also pretty obvious that the models have some built in prompt
| system rules that makes the final output a certain style. They
| seem very consistent
|
| It also looks like 40 has the temperature turned way down, to
| ensure max adherence, while midjourney etc seem to have higher
| temperature.more interesting end results, flourishing, complex
| Materials and backgrounds
|
| Also what's with 4o's sepia tones. Post editing in the gen
| workflows?
|
| I don't believe any of these just generate the image though,
| there's likely several steps in each workflows to present the
| final images outputted to the user in the absolute best light.
| phi-go wrote:
| There are numbers on how many tries it took. I would also find
| the individual prompts and images interesting.
| simonw wrote:
| You can run some image models locally if you want to prove to
| yourself how well they can do with just a single generation
| from a prompt with no extra steps.
|
| I've done this enough to suspect that most hosted image models
| don't increase their running costs to try and get better
| results through additional passes without letting the user know
| what they are doing.
|
| Many of the LLM-driven models do implement a form of prompt
| rewriting though (since effectively prompting image models is
| really hard) - some notes on how DALL-E 3 did that here:
| https://simonwillison.net/2023/Oct/26/add-a-walrus/
| thorum wrote:
| Actual link seems to be: https://genai-showdown.specr.net/image-
| editing
| typpilol wrote:
| This is the editing link yes. I just got done looking at it
| from the other link.
|
| The other stuff is text to image (not editing)
| neilv wrote:
| > _" A dolphin is using its fluke to discipline a mermaid by
| paddling it across the backside."_
|
| If this one were shown in a US work environment, I might say a
| collegial something privately to the person, about it not seeming
| the most work-appropriate.
| PieTime wrote:
| I think I'd probably say that the prompts are telling me more
| about the author than I think is necessary for these tests... I
| hope they were at least sampled from responses.
| echelon wrote:
| Please fix the title, or change the link.
|
| The title of this article is "image editing showdown", but the
| subject is actually prompt adherence in image generation from
| prompting.
|
| Midjourney and Flux Dev aren't image editing models. (Midjourney
| is an aesthetically pleasing image generation model with low
| prompt adherence.)
|
| Image editing is a task distinct from image generation. Image
| editing models include Nano Banana (Gemini Flash), Flux Kontext,
| and a handful of others. gpt-image-1 sort of counts, though it
| changes the global image pixels such that it isn't 1:1 with the
| input.
|
| I expect that as image editing models get better and more
| "instructive", classical tools like Photoshop and modern hacks
| like ComfyUI will both fall away to a thin fascade over the
| models themselves. Adobe needs to figure out their future,
| because Photoshop's days are numbered.
|
| Edit: Dang, can you please fix this? Someone else posted the
| actual link, and it's far more interesting than the linked
| article:
|
| https://genai-showdown.specr.net/image-editing
|
| This article is great.
| jumploops wrote:
| Slight nit: it lists "OpenAI 4o" but the model used by ChatGPT is
| a distinct model labeled "gpt-image-1" iirc
|
| A prompt id love to see: person riding in a kangaroo pouch.
|
| Most of the pure diffusion models haven't been able to do it in
| my experience.
|
| Edit: another commenter pointed out the analog clock test, lets
| add the "analog clock showing 3:15" as well (:
| ZiiS wrote:
| The link is to the imagegen test not the editing one. Here 4o
| was used to preprocess the prompt.
| snailmailman wrote:
| There isn't a date in the article, but I _know_ I had read this
| months ago. And sure enough, wayback has the text-to-image page
| from April.
|
| But the _image editing_ page linked at the top is more recent,
| and was added sometime in September. (And was presumably the
| intended link) I hadn't read that page yet. Odd there is no
| dates, at first glance one might think the pages were made at the
| same time.
| jonplackett wrote:
| Yeah this is very old. Although anything older than a week is
| reasonably old in AI.
| foofoo12 wrote:
| > There isn't a date in the article
|
| SEO guys convinced everyone that articles without dates do
| better on search engines. I hope both sides of their pillow is
| hot.
| ljlolel wrote:
| I discovered this independently myself a decade ago since
| it's true
| master-lincoln wrote:
| fucking marketing people screw us over on so many levels...
| greatgib wrote:
| Gpt4o shows the huge annoyance of the company/model being a moral
| judge of your requests and refusing quite often for anything
| negative.
|
| It's like 1964 but corporate enforced. Now there are tasks that
| you are not allowed to do despite being legal.
|
| In the same way, using gpt5 is now very unbearable to me as it
| almost always starts all responses of a conversation by things
| like: "Great question", "good observation worthy of an expert",
| "you totally right", "you are right to ask the question"...
| holoduke wrote:
| Try some of the Chinese models. Much less restrictive. With
| some obvious exceptions.
| ACCount37 wrote:
| People gave Altman shit for enabling NSFW in ChatGPT, but I see
| that as a step in the right direction. The right direction
| being: the one that leads to less corporate censorship.
|
| >In the same way, using gpt5 is now very unbearable to me as it
| almost always starts all responses of a conversation by things
| like: "Great question"
|
| User preference data is toxic. Doing RLHF on it gives LLM
| sycophancy brainrot. And by now, all major LLMs have it.
|
| At least it's not 4o levels of bad - hope they learned that
| fucking lesson.
| Lerc wrote:
| I have seen a few normally progressive types act quite
| conservative puritan over the NSFW ChatGPT thing. It seems
| there are quite a lot of people consider things to be
| uniformly good or bad and their opinion of the whole colours
| their opinion of the parts.
|
| OpenAI are in a difficult position when it comes to global
| standards. It's probably easier to see from outside of the
| United States, because the degree to which the historical
| puritanism has influenced everything is remarkable. I
| remember the release of the Watchmen film and being amazed at
| how pervasive the preoccupation with a penis was in the media
| coverage.
| thinkingtoilet wrote:
| Name me one piece of enterprise software that lets you do NSFW
| things. The way people jump to 1984 with no thought is double
| plus bad. ChatGPT is a piece of enterprise software. They are
| trying to sell it to large companies at large prices. This is
| not a rhetorical question, do you think if you could generate
| nude images of celebrities or picture of extreme violence,
| corporations would buy it? Having been a director at a Fortune
| 500 company that bought software, I can tell you with 100%
| certainty the answer is "no".
| RobotToaster wrote:
| > Name me one piece of enterprise software that lets you do
| NSFW things.
|
| Photoshop, MS word.
| drdeca wrote:
| Technically? Microsoft Word certainly lets one write smut,
| and Photoshop certainly allows one to draw pornography? They
| won't like, produce NSFW things automatically of course.
| ryandrake wrote:
| Exactly. Programs that don't let you do things based on the
| content _should_ be thought of as weird /broken.
|
| Imagine if we woke up tomorrow morning and grep refused to
| process a file because there was "morally objectionable"
| content in it (objectionable as defined by the authors of
| grep). We would rightly call that a bug and someone would
| have a patch ready by noon. Imagine if vi refused to save
| if you wrote something political. Same thing. Yet, for some
| reason, we're OK with this behavior from "certain"
| software?
| pants2 wrote:
| Companies like PH use full Enterprise stacks from AWS to
| Oracle. Hell, CloudFlare actively takes flack for running
| much worse websites like 8Chan, Daily Stormer, etc. and they
| are as enterprise-focused as it gets.
| addend wrote:
| Is there any AI image generator/editor that is good at creating
| graphics with transparent background? Nano Banana and some others
| output a white grey checkered background (fake transparency).
| neurostimulant wrote:
| There is https://leonardo.ai/transparent-png-generator/
| tezza wrote:
| If you're interested in side by side analysis of carious image
| gen tools, i review them:
|
| https://generative-ai.review/2025/09/september-2025-image-ge...
| indigodaddy wrote:
| EDIT, looks like I didn't click on the "image editing" tab when I
| went to the site, so I guess take the rest of my below comments
| criticizing the terminology with a grain of salt...
|
| "Image editing" is a curious term, as it appears the site/topic
| is actually all about generating new images. The term in my mind
| should be for actual editing of existing, real, images, Eg
| "remove the coffee table" from this living room photo after
| uploading the image. I've found the actual "image generation"
| models to be bad at this because they introduce too many
| artifacts that weren't in the original, which makes sense because
| they are really geared for creating images out of thin air.
|
| Multimodal models like qwen3-vl-30b-a3b, however, seem to do
| quite well with editing existing images without trying to
| constantly add in new things or trying to change the image in
| ways that you don't want, as if it's trying to do the "lets just
| generate a new image" thing. imagegpt.com is also good for
| editing existing images, but not sure what model they are using
| on the backend.
| biinjo wrote:
| I don't know if you and I are looking at the same site because
| all I see is existing images being edited with GenAI.
|
| Input: bald man Prompt: give bald man hair Output: edited
| original, now with hair
|
| That looks like editing to me.
|
| Or are we strictly adhering to the 'generating new images'
| definition because these models technically recreate the entire
| image? It would be like editing a photo in Photoshop. If you
| hit "Save" you edited the photo. But if you hit "Save As" and
| create a new file, the photo wasn't edited but created as a new
| image?
| vunderba wrote:
| I've actually gotten this comment a couple of times - perhaps I
| should make the nav bar at the top more prominently displayed.
|
| WRT to Qwen3, is it possible that the API/site you were using
| was passing your "image edit requests" to something like Qwen-
| Edit [1] under the covers?
|
| To my knowledge, Qwen3-VL (Vision Language) isn't capable of
| generating/modifying images - it's purely for doing reasoning
| about images.
|
| [1] https://huggingface.co/Qwen/Qwen-Image-Edit
| dangoodmanUT wrote:
| To me this goes to show how far ahead Google is in the space.
|
| The ability to clearly understand the image being edited, and
| make edits that look natural to that understanding, are far
| beyond any of the other models.
| cristaloleg wrote:
| Where is the famous "Don't draw a green elephant?"
|
| UPD: suggested here https://github.com/scpedicini/genai-showdown-
| public/discussi...
___________________________________________________________________
(page generated 2025-10-26 23:01 UTC)