[HN Gopher] Implementation of Imagen, Google's text-to-image neu...
___________________________________________________________________
Implementation of Imagen, Google's text-to-image neural network, in
PyTorch
Author : CesarERM
Score : 343 points
Date : 2022-05-26 03:05 UTC (19 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| TekMol wrote:
| What is the reason Google published their research details about
| Imagen?
|
| Why don't they just keep their findings to themselfes and build
| products on top of them?
|
| Public companies can't do stuff just for the fun of it, right? So
| there must be some commercial reasoning behind it?
| toxik wrote:
| It's simply flag planting. If they don't do it, some other
| player will.
| mzs wrote:
| You realize this doesn't include the trained data, right?
| pyinstallwoes wrote:
| Absolute power, corrupts absolutely. Perhaps it's a game
| theoretic approach to leveling the playing field.
| yaroslavvb wrote:
| Researchers like to talk about and show off their work outside
| the company. If you don't let them, they get unhappy and leave.
| minimaxir wrote:
| Publishing high impact research gives credibility to the ML
| teams, which helps recruiting and prestige.
|
| It's less cynical, more incentive alignment.
| hackernewds wrote:
| Also good for society. Less evil, more nice people. Respect
| Google for these traits.
| benoketamad wrote:
| Oh, they don't even want to create/release a usable image
| search... Respect, of course.
| Filligree wrote:
| Didn't they run into legal trouble with that?
| Geonode wrote:
| They could have won that fight, but they don't care
| enough. I use Yandex for images now, it's very good.
| lostmsu wrote:
| Care to link?
| 29athrowaway wrote:
| Because the competitive advantage is also in the training
| datasets and ML infrastructure.
| nl wrote:
| > Public companies can't do stuff just for the fun of it,
| right? So there must be some commercial reasoning behind it?
|
| Yes they can actually.
|
| If you are a shareholder you can either sue (unlikely to
| succeed) or vote against the board. That's pretty much the only
| recourse.
| pmoriarty wrote:
| _" If you are a shareholder you can either sue (unlikely to
| succeed) or vote against the board. That's pretty much the
| only recourse."_
|
| Or you could sell or just threaten to sell your shares.
|
| Buying more of the company's shares to take it over is
| another option.
|
| Good luck doing that with Google.
| DeathArrow wrote:
| They did it out of goodness of their hearts. :)
| blueblob wrote:
| I think it is because the proprietary part for them is the
| data, not the particular algorithm. They benefit more from
| other people making advances on their technology because they
| have the data to get more benefit than anyone else. If they
| kept it to themselves, they would get no "free" advancement. So
| they trade-off the secret of the technique in the hope that
| others will advance the technique, making their data more
| valuable.
| dotnet00 wrote:
| I think there are three main reasons why:
|
| - being open is kind of just how things in ML generally work
| right now, it's in stark contrast to things like chemistry or
| physics where paywalls are pretty common
|
| - it's a matter of clout, ML is moving ridiculously quickly,
| with work from just 5 years ago being considered outdated in
| terms of capability, if you don't publish, someone else will
| and they'll get the credit. This likely also matters for the
| researchers since they get credit too. In a sense this is just
| publish or perish culture from academia.
|
| - it's also somewhat about hiring, which is related to the
| clout. By putting out this kind of research, they're attracting
| talented engineers to consider working for them. This of course
| is pretty relevant to the rest of their business, especially
| given how heavily Google leans on AI to handle moderation.
| nmfisher wrote:
| Two reasons: 1) Even though it's all technically very
| impressive, so far there's not a huge amount of
| commercialization potential here. OpenAI is charging for its
| GPT-3 model but its revenue is probably negligible next to the
| hardware costs (sunk + ongoing) to train it in the first place,
| let alone the researcher salaries they're paying 2) Most of the
| stunning examples are cherry-picked. These things fail much
| more often than they are willing to admit, and they're probably
| assuming (correctly) that not enough people are willing to pay
| for something that only-sorta-kinda-works 1/3 of the time, when
| you're holding it the right way.
| urthor wrote:
| It's also because these companies hire away staff all the
| time.
|
| Rule 1 of working in this field, recruit a mid level member
| of the other company's research group biannually to get the
| latest gossip.
|
| It's impossible to keep a 1 page or shorter "algorithm"
| secret, when the creators are geniuses and they hop jobs
| every year or so.
|
| Fellas with more IQ than games in a baseball season, they
| just don't forget.
| adamhowell wrote:
| I'm currently working fulltime on AI-powered design suite
| Accomplice (https://accomplice.ai) and if you ask me on a
| good day I would tell you I do think there's already huge
| commercial potential. On a bad day, though ;)
|
| My current approach is a "model marketplace"
| (https://accomplice.ai/models) where the most popular open
| source text-to-image models (VQGAN+CLIP, Disco Diffusion,
| DALL-E Mega coming soon...), sit alongside the most popular
| open source style transfer models, and then finally I have
| the ability for a user to finetune their own models using a
| simple drag-and-drop tool (https://accomplice.ai/no-code-
| model-training).
|
| Using this approach a user has enough models to try or train
| that they can have a higher hit rate. For example, Accomplice
| currently has finetuned models for photo realistic people (ht
| tps://accomplice.ai/models/f58bfa91-bb18-406f-a0e1-db00fcf...
| ), watercolor backgrounds (https://accomplice.ai/models/91b8a
| 080-faca-4ff4-8b11-64b0789...), etc...
|
| So theoretically if there were a searchable marketplace of
| 100s of different finetuned models people could choose from,
| they would use it much like an iStockPhoto and be able to
| create the kind of images they want instead of just
| downloading them.
|
| But it's of course a constant work in progress. Slowly
| growing though and lots of promising stuff ahead!
| nautilus12 wrote:
| Why would people do this when aspiring artists are
| practically giving their real photos and paintings for free
| on places like deviant art? Why further commoditize
| something that's already been commoditize to practically
| free?
| adamhowell wrote:
| Well, I'm focusing on business use cases - stock images
| and how Accomplice could be useful for marketing and
| content creation.
|
| i.e. The ability to easily take your logo and stylize it:
| https://accomplice.ai/@adam/iterations/2bcc90ad-3237-486a
| -8d...
|
| Create a photorealistic avatar whenever you need it: http
| s://accomplice.ai/@adam/iterations/988b7d54-dc39-43b1-b5.
| ..
|
| Easily remove the background of a photo:
| https://accomplice.ai/models/97746c4b-c6f0-49cb-
| ae1b-859716b...
|
| Upscale a photo: https://accomplice.ai/models/bd4619ee-82
| 02-4cf0-a04e-291820f...
|
| Etc etc. AI can make all this stuff easier. And you have
| a sense of ownership over what you create. All in one
| place where you can collaborate on all of it with your
| team. I feel like that's valuable. It's certainly a tool
| I've always wanted.
|
| But, also, as a bit of an aside - if the Googles and
| OpenAIs of the world are just going to bite every
| artist's style anyway with a mostly black box service and
| training set... it feels like the option for an artist to
| train/finetune their own model, promote it and possibly
| make money off of that is worth trying.
| pmoriarty wrote:
| I'm an artist, and I'd absolutely love to use something
| like this to inspire me or to give me something to
| continue working with on my own.
|
| In one sense it's kind of like a much "smarter" photoshop
| filter, where it can make your own art/photos look more
| like what you want (ex: Van Gogh, Dali, Picasso, or
| combinations of those, or something completely
| weird/new/different).
|
| You could also train the models on your own work and have
| it generate art in your own style that could inspire you
| or could be useful to you either as a base to work from
| or that you could take interesting elements from to
| create new art.
|
| Similar things can be done in music, by the way, and that
| would be really useful to musicians too.
|
| Poets could use something like this to create poetry,
| novel writers to write novels, etc..
|
| This is really an improvement on the collaboration
| potential between humans and computers -- which is
| probably why it's called "Accomplice".
| nmfisher wrote:
| I meant commercialization potential for companies like
| Google, where anything less than a hundred million is
| probably a failure :) Hopefully for the non-Googles of the
| world (i.e. you), there's a good pathway forward!
| 1024core wrote:
| Your headshots model's outputs look creepy. The eyes are
| off; e.g. the young african girl. You may want to tune your
| loss functions to fix that.
| TekMol wrote:
| I wanted to try your site, but after clicking on the link
| in the email, it tries to send me through some
| redirectingat.com link which is blocked by my ad blocker.
|
| What does that link do?
| adamhowell wrote:
| Update: Ah, found the "Tracking" setting in Sendgrid that
| I thought I had already turned off. Off for sure now.
| Thanks again for the heads up!
|
| The confirmation link should only be going to
| accomplice.ai unless Sendgrid is doing some link tracking
| that I've just forgotten about. Could you forward that
| email to adam at accomplice dot ai if you get a chance.
| Thanks for letting me know!
| jermaustin1 wrote:
| That will get turned back on automatically in an
| "update".
|
| Same thing happened to me multiple times across multiple
| platforms: SendGrid, Mandrill, MailJet, MailGun. I always
| turn off the tracking (enabled by default on all of
| them), but magically its back on a few weeks/months
| later. I've given up finding a solution and just revisit
| my settings every few months to check on it.
| gbasin wrote:
| Cool project! Will follow along
| gfodor wrote:
| Seems exactly false. DALL-E 2 seems to end much of the
| illustrator industry and if the endless array of Twitter
| posts from early adopters are any indication it works great.
| curiousgal wrote:
| What would the product be? My experience is that most ML papers
| are very brittle, for every cool result/example you see there's
| a plethora of nonsense spit by the models.
| frozenport wrote:
| 1. This finding is hard to monetize, especially with ROI that
| Google typically does (for example a an app that makes $500 a
| month isn't worth it)
|
| 2. Deploying models in a cost effective way is hard
|
| 3. Lessons learned from building this model can indeed be
| monetized and many of them may be kept secret.
| dekhn wrote:
| Google started publishing for several reasons but the primary
| one was recruitment (showing off was a secondary goal). The
| mapreduce, GFS, and bigtable papers played an important role in
| attracting an early generation of distributed computing/high
| performance computing people from around the valley and
| CMU/MIT, who helped build the second really successful versions
| of the web search engine (retrieval and ranking), ads serving
| (the auction, the logs joining pipeline), etc.
|
| The other reason is that the leaders at Google at the time
| believed that we would achieve the singularity faster if Jeff
| Dean periodically sent ideas back 10 years in time to Doug
| Cutting.
| quickthrower2 wrote:
| It could be classed as a way to attract talent. It is basically
| then a very expensive fusball table.
| jonathankoren wrote:
| Of course the implementation isn't the issue. It's the training
| data and the compute machines.
|
| Open source is pretty meaningless here
| emadm wrote:
| Images actually used our open LAION 400m dataset plus 400m of
| their own.
|
| On compute there is more than enough compute available to open
| source now via LAION and Eleuther AI to train these models,
| will just a bit of time.
| jfoster wrote:
| In fact, this kind of reverses things, doesn't it?
|
| Open source is built on the assumption that you can do more
| with source code than with binaries. In the case of AI models,
| the computed weights of models are what's valuable, and the
| source code used to achieve them is less useful.
| aero-glide2 wrote:
| How much would it cost in training to match dall-e 2?
| abrichr wrote:
| https://twitter.com/alexjc/status/1347458546636619778?s=21&
| t...
|
| > The blog post says 256 GPUs for 2 weeks, so:
|
| > DALL*E would cost $131,604 to train on AWS, assuming a
| p3.16x-large at market rates. Could be as low as $40k if
| you already paid for reserved instances.
| dang wrote:
| Recent and related:
|
| _DALL-E 2 open source implementation_ -
| https://news.ycombinator.com/item?id=31228710 - May 2022 (152
| comments)
|
| Also:
|
| _X-Transformers: A fully-featured transformer with experimental
| features_ - https://news.ycombinator.com/item?id=27089208 - May
| 2021 (37 comments)
|
| _Text to Image Generation_ -
| https://news.ycombinator.com/item?id=26615791 - March 2021 (88
| comments)
| tomatowurst wrote:
| What's it going to take to train this on porn? This is something
| that can be crowdfunded.
|
| Look deepnude is a thing and somebody is making money off it:
| https://app.deepnude.cc/upload
| chirau wrote:
| Lol. Not the same thing obviously but your question just
| reminded me of @robotpornaddict, a neural net that watches porn
| and tries to describe what it sees.
|
| https://twitter.com/robotpornaddict
| hackernewds wrote:
| They specifically mention in the Images documentation that one
| reason they haven't publicized this yet is due to porn or deep
| fakes risk. Assuming this works very well already.
| 29athrowaway wrote:
| How can this be the most upvoted comment?
|
| What you describe is already illegal on many jurisdictions.
| lajamerr wrote:
| Which part is illegal?
|
| Are you talking about porn? Or using potential copyright
| material in an infringing manner? Or perhaps the possibility
| of "deepfake" porn?
| 29athrowaway wrote:
| In some jurisdictions, deepfakes of any kind are treated as
| a form of defamation. In others, adult deepfakes
| specifically are illegal.
| pmoriarty wrote:
| Which jurisdictions are those?
| tomatowurst wrote:
| Korea. Hardcore pornography is also illegal there.
|
| Keep in mind this is a country where if you leave a bad
| review after you get scammed by someone with evidence, it
| is defamation. So not quite leadership the world needs in
| this industry.
|
| Really sad to see ppl on HN flagging all of my comments
| on this thread. I mean it's not like you can't find
| celebrity deepfakes including Kpop.
|
| The cat is out of the bag and its only going to get
| better and faster from here whether some
| cultures/jurisdictions take offense or not.
| DeathArrow wrote:
| And what is going to take to generate videos, not pictures?
| a2800276 wrote:
| Maybe if you study hard enough, one day you'll have what it
| takes to make a real girl.
|
| https://youtu.be/9qd04u2Yj44
| tomatowurst wrote:
| To run we must learn to walk first. To walk we must learn to
| be erect. To be erect we must master crawling.
|
| I don't think the leap is too crazy if we are talking short
| moving pictures without sound. However, when sound gets
| involved, this is where it would become very tricky.
| pmoriarty wrote:
| _" when sound gets involved, this is where it would become
| very tricky"_
|
| I don't think a neural net would have much trouble
| generating moans in sync to the motion.
|
| Why pay humans for all that fake moaning when an AI could
| do it?
| tomatowurst wrote:
| actually you are right, sound generation here wouldn't be
| as hard as I originally thought.
| comex wrote:
| porn_rec_throw wrote:
| [deleted]
| londons_explore wrote:
| This tech is actually pretty bad at human faces - it all
| appears unnatural and distorted. You get things like the
| occasional extra earlobe...
| Al-Khwarizmi wrote:
| Human faces were excluded on purpose from the DALL-E 2
| training set in order to prevent misuse. I suppose the same
| will be the case here (or at least, in public-facing versions
| of the models).
|
| Given how it renders dog faces, I don't see why it wouldn't
| be good at human faces too, if trained for it.
| jstummbillig wrote:
| While the "it will never do as well as humans at this one
| thing I feel strongly about" bias is still a common one, even
| on hn, at this point I am fairly certain that we will soon
| all have to live with the fact that we are no longer all that
| special in regards to, well, absolutely everything.
|
| One of my more interesting realisations over this development
| is that being human is apparently a religion to a lot of
| otherwise secular humans.
| pmoriarty wrote:
| _" we are no longer all that special in regards to, well,
| absolutely everything"_
|
| If/when general AI comes about, maybe so.
|
| Until then all we've got are a bunch of highly specialized
| tools/helpers/slaves that may be good (in some sense) at
| one thing and awful at pretty much everything else.
|
| You could argue that humans themselves are an ensemble of
| such highly specialized parts that are more than the sum of
| their parts in some ineffable way that's more of a "we know
| it when we see it" than something that's formalizable.
| Machines/computers lack that... so far.
| eminence32 wrote:
| I wonder if this implementation was trained on the same dataset
| that trained DALL-E2, would it produce results of comparable
| quality.
| tomatowurst wrote:
| I have to clarify I am visioning DALL-E-2-PORN.
|
| Couldn't you just scrape porn to get copious amount of dataset?
| Scope would be narrower and thus require less classification
| and in general it has common themes.
|
| I'm more concerned how expensive it will be to train it on GPU
| instances. We are looking at A6000s right? That's like $15/hr.
| Karawebnetwork wrote:
| Sure, you could use images you do not have the legal rights
| to if you do not release anything at the end / just leak the
| final result. But this is a lot of effort and resources
| invested for something that would effectively never be shared
| on the clear web.
| ghotli wrote:
| "just"
|
| All the complexity wrapped up into one word.
| tomatowurst wrote:
| You do realize web scraping has gotten very cheap at scale
| and easy now too. It's an after thought for me, I'm more
| concerned with the economics of training, it can't be cheap
| visarga wrote:
| You don't realize how hard it is to make an inclusive,
| cleaned up dataset. Take a look at this Notion from
| BigScience detailing their workgroups. Three of them are
| related to preparing the dataset.
|
| https://bigscience.notion.site/10743770aae24ff3bdc1b938cf
| 454...
|
| And this is just for a text-only scrape.
| dotnet00 wrote:
| In my experience, scraping the data is the easy part.
| Once you've scraped it you've got to get rid of all the
| garbage, which is where the issues arise, especially if
| you're just blindly scraping everything you can find.
|
| For example, in a generative model I'm working on, I have
| a dataset consisting of ~5M images just blindly scraped
| from a website. After filtering, this drops down to ~500k
| images, yet a model trained on that performs worse than
| one trained on a set of 30k curated images (picked based
| on a manually evaluated list of "known good" artists),
| where filtering brings it down to ~18k images. The larger
| dataset, while containing more information, also contains
| more errors, many of them pretty hard to filter out.
| tomatowurst wrote:
| Which is why porn is such a great dataset for
| crowdsource:
|
| - lots of people are stimulated by it
|
| - lots of people want DALL-E-2 for porn
|
| - and lots of people are willing to work towards that
| common goal
|
| The beauty of this is that people are just going to keep
| coming and coming to it.
|
| Like I'm trying to be mature and serious about this.
| What's it going to take?
|
| - Community responsible for scraping dataset, generating
| image dataset from moving pictures, upscaling said
| dataset.
|
| - Crowdsourced labeling, cleaning, normalizing dataset
|
| - Crowdfunding to train, host, and publish.
|
| All of the above are not easy by any means but its much
| more achievable than trying to build a generic DALL-E
| that can create anything.
|
| Really my point is that the scope of the dataset has
| narrower range in terms of desired output where as
| DALL-E-2 casts a far wider net.
|
| Specialization is key here.
| dotnet00 wrote:
| I was writing up a whole explanation for how that
| couldn't possibly work, but tbh you aren't really wrong
| with that, after all captchas these days do similar work.
|
| I guess the primary concern with a porn model would be
| the ethics of it, which might turn off any company from
| helping out on training resources (for example, Google's
| TPU Research Cloud requires you to follow their code of
| ethics on AI, which would be very difficult to do with a
| topic as sensitive to people as porn).
| fluoridation wrote:
| Sincerely asking: why is this an issue of ethics? What
| about algorithmically generating porn would be unethical?
| turtledove wrote:
| You are going to need to figure out how to filter out
| copyrighted works and images for which you don't have
| permission (eg, someone uploading a picture of an ex
| partner).
| tomatowurst wrote:
| If its in public domain, there is no implicit right to
| privacy. It's going to be super difficult to claim
| derivative of copyrighted material going forward.
| turtledove wrote:
| Sure, but how much of that material is in the public
| domain? I'd assume not a lot.
| mpalmer wrote:
| Web scrape... what, exactly? How will "web scraping" lead
| to high quality results?
| visarga wrote:
| Already does. It works by magic, we put images on one
| side and text on the other side and then power it up. Let
| it simmer for a few months at megawatt power levels.
| Kleto wrote:
| Reddit has plenty of nsfw categories.
|
| And tags.
|
| Insta has tags.
|
| Plenty of others have nsfw and tags
| tomatowurst wrote:
| if you have to ask, the answer would be irrelevant.
| mpalmer wrote:
| I certainly don't want to waste the time of someone who
| knows what they're talking about.
| minimaxir wrote:
| Training an AI requires high quality, clean/normalized
| data, which is very difficult to do at millions/billions
| of data points and is _very_ frequently done incorrectly
| with silent failures.
| tomatowurst wrote:
| There's a section of the internet where you can easily
| find _billions_ of images or can be generated from moving
| pictures. Even upscaled. I really didn 't expect to have
| to spell it out.
|
| hint: they are all about one thing and there are a lot of
| eager volunteers to help on those websites. It would be
| easy to "normalize/clean/classify" because the pictures
| would have a consistent theme, thus reducing the amount
| of parameters.
|
| We are not trying to generate elephants getting railed on
| a SpaceX rocket flying in oil painting style (although
| I'm sure theres people into that and its not my place to
| judge), we are just trying to remove the human cost out
| of this necessary evil.
|
| I can't believe nobody is investing in "DALL-E-2-4-PORN".
| This sounds like an X amount of money thrown at a hugely
| sticky product that can be iterated (with the current
| trend in hardware) to the point where it literally
| generates billions of dollars in revenues for ages to
| come (no pun intended).
| thorum wrote:
| Does anyone actually want porn of fake, AI generated
| "people"? Seems like most of the demand would be
| deepfakes of real people, which is both highly unethical
| and a good way to get your business sued out of
| existence.
| sterlind wrote:
| furry porn? vore? transformation? selfcest? there's a ton
| of niche fetishes that aren't really doable IRL, and only
| exist as art. generated images would be a game changer.
| Kleto wrote:
| Why not?
|
| Do you prefer known porn actress over non known ones?
|
| I would say that I might know a handful of names but
| search for them rarly.
| aqme28 wrote:
| There's also seemingly unlimited content of real people
| produced consensually by those people. What can DALL-E
| generate that you can't find on pornhub or onlyfans??
| Kleto wrote:
| They cost money.
|
| They involve real woman (ethical concerns).
|
| They might not show exactly what you are looking for.
|
| Like the good scene is to short or the quality is too bad
| or there is only one video.
|
| Also variation.
|
| ML porn should be a good thing. Only thing Im not sure is
| about people creating pedo porn. But even that is better
| than real pedo porn :|
| badwolf wrote:
| https://davidmack.medium.com/what-i-learned-from-
| building-an...
| 29athrowaway wrote:
| I guess the stock photos industry will be disrupted by this.
| karmasimida wrote:
| How big is getty? I think this is a billion dollars business.
| seydor wrote:
| They can make more profit if they don't need to pay artists
| benstrumental wrote:
| Definitely. With the right training set, video game assets too.
| Imagine being able to generate N variations of any asset in any
| video game style...
| quickthrower2 wrote:
| r/wtfstockphotos will be full of this
|
| And medium posts about React hooks and Go generics are gonna be
| full of "Animated Drake meme but with Rick Astley" kind of fun.
| throwaway12245 wrote:
| I did the pip installs and installed Cuda. I changed prompt and
| ran the sample code. It ran to completion. How do I save the
| image from trainer.sample?
| visarga wrote:
| You'd have to train it first.
| [deleted]
| bufferoverflow wrote:
| How much would it cost to train something like this? Is there
| even a good dataset for it?
| dirtyid wrote:
| I feel like the fact big porn hasn't poached talent and jumped
| all over this suggest at least 10s of millions. That said some
| for profit no-rules deepfake service for disinformation and
| illegal content has to be in the works.
| tomatowurst wrote:
| There's a company in Montreal that makes that in a month and
| also has access to copious amount of said datasets on their
| servers. It may or may not be that they already have
| engineers on it. We have no way of knowing since its a
| private company
| Yenrabbit wrote:
| There is a dataset of 5 billion image-text pairs (laion-5b)
| scraped by various parties. This can then be filtered and used
| to train these models. Cost is a bit of an issue but there are
| orgs that have provided compute for open model training. And
| Imagen is nice because the text encoder part is already
| available and doesn't need more training, so it would just be
| the diffusion model components being trained. I'd guess we'll
| see a biggish training run starting in a few weeks.
| visarga wrote:
| I hope so. It's a bit cruel to show off and then lock it
| away.
| dharma1 wrote:
| When you say a bit of an issue, how much are we talking?
| Yenrabbit wrote:
| Four or five figures I'd guess? I'm not clued up on
| costs/performance for TPU stuff to give a better estimate,
| but guessing at a week on a 256 TPU pod, call it $30k?
| Voloskaya wrote:
| You are off by an order of magnitude at least. 256
| TPUs-v4 (not pod), would cost you around 20k$/day. They
| actually used 512 TPUs (256 for base model + 128 for each
| of the two superresolution models).
|
| Assuming an average training time of 1 week as you said,
| that gives us about 280k$.
|
| It's also most likely trained for longer than a week, the
| base model for Dalle-2 was trained for 100-200k GPU
| hours, so between 2-4x longer than that, we can guess
| this is roughly similar.
|
| You also never successfully train everything first try,
| so all in all, to replicate this work just from the
| paper, we are talking about at least 500k$.
| rockemsockem wrote:
| While that is what they did, they also used a batch size
| of 2048 while training. This is just to speed training
| up, not a hard requirement. It's easy for Google to
| justify more money on compute to save engineer iteration
| loops.
|
| I'll have to read the paper for more details, but it
| would almost certainly cost less (and take longer) to
| train a model like this in a more resource constrained
| situation than Google faces .
| tomatowurst wrote:
| Couldn't a bunch of us shell out $5000~$50,000 and do
| this ourselves? Create a non-profit shell corporation
| outside US jurisdiction, issue shares, raise funds and
| open source the result?
|
| The shares would simply be votes towards future training
| dataset endeavors as no profit would be booked here. Say
| you buy 5000 out of 500,000 shares, that would give you
| 1% voting power in what dataset to train.
| joshvm wrote:
| TF Research Cloud. Especially if multiple people could
| use it with eg model checkpoints:
|
| > 5 on-demand Cloud TPU v3 devices, 5 on-demand Cloud TPU
| v2 devices, and 100 preemptible Cloud TPU v2 devices for
| free for 30 days
|
| So up to 7k hours on demand and 70k pre-emptible
| bufferoverflow wrote:
| Go right ahead, spend $5-50K of your own money and let us
| use it for free.
| bergenty wrote:
| Who is going to be liable for the flood of child
| pornography that comes out of that setup?
| tbalsam wrote:
| Good to see LucidRains get the love he (rightly) deserves. He's a
| beast!
|
| As a thank you to him -- he also does for work for
| commission/etc, check hus GitHub page for more info. I'm not
| fiscally or currently otherwise directly linked to him too
| closely, I've just hung around a while and think he deserves far
| more credit than he gets. This is literally the smallest piece of
| the pie of what the man does across several subdisciplines, send
| him a thank-you please, if possible!
| [deleted]
| emadm wrote:
| Second this, one of the biggest contributors to open source AI
| and someone everyone should sponsor.
___________________________________________________________________
(page generated 2022-05-26 23:02 UTC)