[HN Gopher] Stretching Each Dollar: Diffusion Training from Scra...
___________________________________________________________________
Stretching Each Dollar: Diffusion Training from Scratch on a Micro-
Budget
Author : lnyan
Score : 47 points
Date : 2025-01-13 09:55 UTC (3 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| philipkglass wrote:
| The differently styled images of "astronaut riding a horse" are
| great, but that has been a go-to example for image generation
| models for a while now. The introduction says that they train on
| 37 million real _and synthetic_ images. Are astronauts riding
| horses now represented in the training data more than would have
| been possible 5 years ago?
|
| If it's possible to get good, generalizable results from such
| (relatively) small data sets, I'd like to see what this approach
| can do if trained exclusively on non-synthetic permissively
| licensed inputs. It might be possible to make a good "free of any
| possible future legal challenges" image generator just from
| public domain content.
| godelski wrote:
| > Are astronauts riding horses now represented in the training
| data more than would have been possible 5 years ago?
|
| Yes.
|
| Though I'm a bit confused why this became the goto. If I
| remember correctly the claim was about it being "out of
| distribution" but I have high confidence that astronauts riding
| horses are within the training dataset prior to DALL-E. The big
| reason everyone should believe this is because astronauts have
| always been compared to cowboys. And... what do we
| stereotypically associate with cowboys?
|
| The second reason, is because it is the main poster for the
| 2006 movie The Astronaut Farmer:
| https://en.wikipedia.org/wiki/The_Astronaut_Farmer
|
| But here's some other ones I found that are timestamped. It's
| kinda hard to find random digital art that is timestamped.
| Looks like even shutterstock doesn't... And places like
| deviantart don't have great search. Hell... even Google will
| just flat out ignore advanced search terms (the fuck is even
| the point of having them?). The term is so littered now that
| this makes search difficult, but I found two relatively
| quickly.
|
| 2014: https://www.behance.net/gallery/18695387/Space-Cowboy#
|
| 2016: https://drawception.com/game/DZgKzhbrhq/badass-space-
| cowboy-...
|
| But even if the samples did not exist, I do not think this
| represents a significantly out of distribution, if at all,
| image. Are we in doubt that there's images like astronauts
| riding rockets? I think certainly there exists "astronaut
| riding horse" along the interpolation between "person riding
| horse" and "astronaut riding <insert any term>". Mind you,
| generating samples in distribution but not in training (or
| test) is still a great feat and impressive accomplishment. This
| should in no way be underplayed at all! But there is a
| difference in claiming out of distribution. >
| I'd like to see what this approach can do if trained
| exclusively on non-synthetic permissively licensed inputs
|
| One minor point. The term "synthetically generated" is a bit
| ambiguous. It may include digital art. It does not necessarily
| mean generated by a machine learning generative model. TBH, I
| find the ambiguity frustrating as there is some important
| distinctions.
| throwaway314155 wrote:
| It wasn't because it was "out of distribution" (although
| that's a reasonable assumption and it is at least _somewhat_
| out of distribution, given the scarcity of your examples).
|
| Like the avocado armchair before it, the real reason was
| simply that it "looked cool". It scratched some particular
| itch for people.
|
| For me, indeed it's correlated with "imagination". An avocado
| armchair output had a particular blending of concepts that
| matched (in my mind) the way humans blend concepts. With the
| "astronaut riding a horse on the moon", you are hitting a
| little of that; but also you're effectively addressing
| criticism about text-to-image models with a prompt that
| serves as an evaluation for a couple of things:
|
| 1.) t2i is bad at people (astronaut)
|
| 2.) t2i struggles with animal legs (horse)
|
| 3.) t2i struggles with costumes, commonly putting the
| spacesuit on both the astronaut _and_ the horse - and
| mangling that in the process (and usually ruining any sense
| of good artistic aesthetics).
|
| 4.) t2i commonly gets confused with the moon specifically,
| frequently creating a moon _landscape_ but also doing
| something silly like putting "another" moon in the "night
| sky" as well.
|
| There are probably other things. And of course this is
| subjective. But as someone who followed these things as they
| happened, which was I believe the release of DALL-E 2 and the
| first Stable Diffusion models, this is why I thought it was a
| good evaluation (at the time).
|
| edit: I truly despise HN comment's formatting rules.
| llm_trw wrote:
| >The estimated training time for the end-to-end model on an
| 8xH100 machine is 2.6 days.
|
| That's a $250,000 machine for the micro budget. Or if you don't
| want to do it locally ~$2,000 to do it on someone else's machine
| for the one model.
| GaggiX wrote:
| You can do it on one single GPU but you would need to use
| gradient accumulation and the training would probably last 1-2
| months on a consumer GPU.
| programd wrote:
| Accepting the 1-2 month estimate at face value we're firmly
| in hobby territory now. Any adult with a job and a GPU can
| train their own models for an investement roughly equivelent
| to a high end gaming machine. Let's run some very hand wavy
| numbers:
|
| RTX 4090 ($3000) + CPU/Motherboard/SSD/etc ($1600) + two
| months at full power ($300) is only on the order of $5000
| initial investment for the first model. After that you can
| train 6 models per year to your exact specifications for an
| extra $150 per month in power usage. This cost will go down.
|
| I'm expecting an explosion of micro-AI models specifically
| tailored for very narrow use cases. I mean Hugging face
| already has thousands of models, but they're mostly reusing
| the aligned big corporate stuff. What's coming is an
| avalanche of infinately creative micro-AI models, both good
| and bad. There are no moats.
|
| It's going to be kind of like when teenagers got their hands
| on personal computers. Oh wait....
| godelski wrote:
| From the abstract Finally, using only 37M
| publicly available real and synthetic images, we train a 1.16
| billion parameter sparse transformer with only $1,890
| economical cost and achieve a 12.7 FID in zero-shot generation
| on the COCO dataset. Notably, our model achieves competitive
| FID and high-quality generations while incurring 118x lower
| cost than stable diffusion models and 14x lower cost than the
| current state-of-the-art approach that costs $28,400.
|
| Figure 1 Qualitative evaluation of the image
| generation capabilities of our model (512x512 image
| resolution). Our model is trained in 2.6 days on a single
| 8xH100 machine (amounting to only $1,890 in GPU cost) without
| any proprietary or billion image dataset.
|
| End of intro under the key contributions bullet points
| - Using a micro-budget of only $1,890, we train a 1.16 billion
| parameter sparse diffusion transformer on 37M images and a 75%
| masking ratio that achieves a 12.7 FID in zero-shot generation
| on the COCO dataset. The wall-clock time of our training is
| only 2.6 days on a single 8xH100 GPU machine, 14x lower than
| the current state-of-the-art approach that would take 37.6
| training days ($28,400 GPU cost).
|
| I'm just saying, the authors are not trying to hide this point.
| They are making it abundantly clear.
|
| I should also mention that this is the most straightforward way
| to discuss pricing. It is going to be much more difficult if
| they do comparisons including the costs of the machines as then
| there needs to be an amortization cost baked in and that's
| going to have to include costs of electricity, supporting
| hardware, networking, how long the hardware is used for, at
| what percentage utility the hardware is, costs of employees to
| maintain, and all that fun stuff. Which... you can estimate
| by... GPU rental costs... Since they are in fact baking those
| numbers in. They explain their numbers in the appendix under
| Table 5. It is estimated at $3.75/H100/hr.
|
| Btw, they also state a conversion to A100s
| __loam wrote:
| The pixel art these models produce continues to look like shit
| and not be actual pixel art.
|
| Where'd you get your dataset? Did you get permission from the
| rightsholders to use their work for this?
___________________________________________________________________
(page generated 2025-01-16 23:00 UTC)