[HN Gopher] Visual ChatGPT
___________________________________________________________________
Visual ChatGPT
Author : debdut
Score : 641 points
Date : 2023-03-10 03:55 UTC (19 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| lwneal wrote:
| The most incredible thing about this system is that it uses
| Stable Diffusion (the open source AI art generator), rather than
| DALL-E (the proprietary closed art generator owned by OpenAI).
|
| The fact that even Microsoft, which partially owns OpenAI, is
| giving up on DALL-E shows the power of building an open-source
| community around models with published, downloadable weights.
| MuffinFlavored wrote:
| So why hasn't there been a "as popular as ChatGPT" open source
| version?
| zamnos wrote:
| Two things stand out. Stable Diffusion; the weights for it
| and the supporting Python code was open sourced and released
| to the public. Anyone can find the cpkt file and the Python
| code online and download it and (with a bit of work) run it
| on their own computer if they have the hardware (any
| reasonable GPU) for it. There has been no such release by
| OpenAI, the closest we got is Facebook's leaked LLama model,
| and that's not a chat bot. So we don't have a model to run.
| Stability AI/Emad paid to train the model, which cost like a
| half million dollars in GPU time (he obviously didn't pay the
| retail price of $600k, but also it's not something you'd get
| right on the first shot either) and then gave the output from
| that away. It's not clear how much it would cost to train a
| comparable chat bot to ChatGPT but the impression is that
| would take much more.
|
| The second thing is that it's not clear that we, the Internet
| at large would actually benefit from the model's release.
| StableDiffusion is 4 gigs and able to run on all sorts of
| consumer grade hardware, leading to such a Renaissance.
| ChatGPT makes liberal use of Nvidia A100 GPUs that are
| available to them to use as compute in Azure. (AWS and GCP,
| along with many AI focused smaller cloud companies also offer
| these.) One of those costs, like, $10k. And you need several
| of them to be able to run ChatGPT. Which means even if OpenAI
| were to live up to their name and release ChatGPT's model,
| only businesses and research labs would actually have the
| hardware to run it, so it would be awesome to have the
| weights, but you wouldn't have the same army of developers
| able to work on it.
|
| There _are_ open source LLM chat bots out there, so I think
| we will see one become popular, but at least that says why it
| 'll be a second before we do.
| narrator wrote:
| I have been fooling around with the small 7B llama models.
| They chat, but they are pretty dumb compared to ChatGPT.
| This means they are terser and they confabulate more, even
| for things that are common knowledge. It seems, from asking
| it questions about current events, that the model was
| trained up to data from early 2020.
|
| I haven't seen much output yet from the biggest 65B
| parameter llama model. One can rent cloud VMs that can run
| it for $1.25 an hour or so on vast.ai to run it, but
| ChatGPT is $20 a month so why bother, unless you like the
| fully uncensored aspect.
| tomohelix wrote:
| Likely training and running cost.
|
| Most AI art generators are run on single GPU and can be
| trained with some top-of-the-line consumer hardware.
| Expensive but accessible.
|
| A full blown LLM like ChatGPT is literally the cost of a
| small startup to build and trained. Running it is near
| impossible without cards like A100 which alone costs more
| than a full enthusiast grade PC.
|
| Maybe eventually they will distill and optimize the models so
| that we can fit these things on a PC, then laptop, and then
| phones. But for now it is exclusively the domain of big tech.
| MuffinFlavored wrote:
| > Likely training and running cost.
|
| Why wasn't this a problem for StableDiffusion vs DALL-E?
| OmarAssadi wrote:
| Stable Diffusion will run on any decent gaming GPU or a
| modern MacBook, meanwhile LLMs comparable to
| GPT-3/ChatGPT have had pretty insane memory requirements
| - e.g.,
| <https://github.com/facebookresearch/metaseq/issues/146>
| brookst wrote:
| Worth noting that the M-series macbooks are UMA so 100GB
| VRAM is costly but easily accessible. Their GPU
| performance is nowhere near a 96GB A100, but for sheer
| VRAM it's a good choice.
| tomohelix wrote:
| They are smaller models with less parameters. Their
| original small sizes relative to LLMs also let people
| play around with it and tune it to run on less expensive
| hardware, if the weights are given, ie open source like
| SD.
|
| Originally SD was quite hard to run, with an 8GB high end
| card only outputting 256x256 images. Then AMD and NVIDIA
| started releasing 16GB and 24GB consumer cards and people
| start doing training on those GPUs and tuning their own
| models. Now we have plenty of cards and models that can
| do 512x512.
| MuffinFlavored wrote:
| > They are smaller models with less parameters.
|
| I wouldn't have guessed image is a smaller model/easier
| to manipulate/generate than text.
| speedgoose wrote:
| DALL-E has given a new given a new "experimental" model to a
| few lucky users, it looks a bit better but it seems to have
| less variety currently. I don't think they will catch back the
| competition. ControlNet is so good and I guess that MidJourney
| 5 and StableDiffusion 3 are going to be fully released before.
| nprateem wrote:
| Millions of images have been removed from the training set
| for SD3 which is why a lot of people are sticking to 1.5.
| speedgoose wrote:
| 1.5 is a lot more stupid than 2.1 in my experience.
|
| Some models based on 1.5 produce good looking images if it
| produces what you asked for, but it's often a miss on more
| complex compositions.
|
| We only start to see good 2.1 models like the Illuminati
| one. I have good hopes about the version 3, and I hope
| people will fine-tune it to their desires (that seems to
| mainly be young looking women with unrealistic bodies).
| ilaksh wrote:
| I also have a website that integrates Stable Diffusion with
| ChatGPT.
|
| https://aidev.codes
|
| Now everyone will go ahead and bury my comment.
| quickthrower2 wrote:
| I get an error when I try to log in - literally the response
| is the text "Error" and nothing else, not even HTML!
| ilaksh wrote:
| I know its not a useful message. Its one of the more
| obvious things to be improved. Sorry about that. I will see
| if I can quickly improve it.
|
| OK I fixed the message -- that actually means invalid
| username.. whoops.. OR invalid password. Lol. What user is
| it?
|
| It will say which error now at least.
| omnicognate wrote:
| You're getting a lot of downvotes and backlash because of the
| way that you're posting. I understand how you feel, though.
| It's very personal if you pour effort into something and
| people aren't interested. It plays with your sense of self-
| worth. It's a nasty feeling. It's easy to be bitter.
|
| I'm not going to check out the site because I'm not
| interested in ML generation of websites, or even manual
| creation of them. I'm here because I'm interested in the
| discussion around LLMs. That doesn't mean it doesn't have
| value, though.
|
| If you're going to follow this entrepreneurial path you need
| to develop a thick skin and learn to cope with rejection.
| You're going to get things wrong a lot, and a large amount of
| your effort will be "wasted" trying things that don't work
| out. You need to learn from your mistakes and understand your
| own strengths and weaknesses (eg. if you're not good at
| marketing, involve someone who is). If you want something
| where your efforts will reliably be rewarded you need to get
| a regular job instead.
|
| I upvoted, fwiw.
| vitorgrs wrote:
| Not giving up sadly, recently they launched bing.com/create.
| swyx wrote:
| > even Microsoft, which partially owns OpenAI, is giving up on
| DALL-E
|
| hold on to your wild extrapolations there. this is a paper by 6
| people from Microsoft Research Asia, which seems based out of
| China. 6 researchoors publishing a thing independently does not
| mean Microsoft "giving up on DALL-E".
| ShamelessC wrote:
| Yeah I agree that GP is hyperbole. Still though, at the very
| least it shows that the researchers found it easier to work
| with weights they could run locally rather than via another
| API call.
|
| I assume this is because DALLE2 still doesn't provide
| embeddings and/or finetuning via API. In addition to likely
| being more expensive to run.
|
| Happy to be corrected on any of this - I still haven't read
| the paper.
| shp0ngle wrote:
| I think they ate using StableDiffusion and not Dall-E? Which
| makes it kind of funny
| hackerlight wrote:
| There are more examples in the paper:
|
| https://arxiv.org/pdf/2303.04671.pdf
| iamflimflam1 wrote:
| Linked paper is available here: https://arxiv.org/abs/2303.04671
| mmq wrote:
| I think the chat interface is a bit restrictive when it comes to
| multimodal models. A much cleaner interface would be an "AI
| notebook" where the user can move, compare, rerun blocks. Also
| sharing, versioning and collaborating with others on notebooks is
| more straightforward.
| kilgnad wrote:
| Now is a really good time to make a start up called skynet.
| qntmfred wrote:
| hmmm can I use this to see how far away we are now
|
| https://karpathy.github.io/2012/10/22/state-of-computer-visi...
| Havoc wrote:
| Happy that this is <8gb vram. Neatly fits into medium/highish
| consumer GPUs
| iandanforth wrote:
| This feels like it owes more to LangChain than a link at the
| bottom of the page.
|
| Compare their prompt:
|
| https://github.com/microsoft/visual-chatgpt/blob/main/visual...
|
| With that of the LangChain ReAct conversational agent:
|
| https://github.com/hwchase17/langchain/blob/master/langchain...
|
| Also it seems appropriate to cite the original ReAct paper (from
| Google mainly)
|
| https://arxiv.org/abs/2210.03629
| LastTrain wrote:
| I think you mean here:
| https://github.com/hwchase17/langchain/blob/master/langchain...
| iandanforth wrote:
| Fixed thanks!
| mcaledonensis wrote:
| Yes, in this integration GPT3 connects with the image space
| only through text and doesn't consume or emit image space
| embeddings / tokens. With GPT4 that perception modality will be
| available directly in the model. And it'll be a better
| integration.
| skybrian wrote:
| How do you know? I've seen multiple claims about GPT4 online
| but people never say where they came from.
| golol wrote:
| future AI systems based on LLMs and other foundation models might
| think less like individuals and more like companies. Ironically,
| LLMs might finally make symbolic AI possible! The way I see it,
| symbolic AI was always missing a small sprinkle of "general
| intelligence" too amooth things out, to grease the gears and
| connect interfaces. I feel like LLMs have that little bit of
| magical "generality" so we can start building "symbolic" AI
| systems which produce work by managing a number of black box
| models. It is like a company: protocols and management structures
| are a sort of symbolic AI that connects black box humans to
| eachother.
| osigurdson wrote:
| I think GPT is super useful but can't seem to eke any value out
| of DAL-E. Yes, it can draw a bear in a business suit on the beach
| well, which is impressive but I can't think of how to utilize
| this.
|
| As an example, I've tried to get it to draw architecture
| diagrams, it draws a few boxes but then places the strangest text
| on those boxes.
| sebzim4500 wrote:
| Yeah, I think simply generating an image from a prompt is of
| dubious value for most usecases.
|
| Inpainting, etc. with SD and control net is really handy for
| image editing though. You can make changes in seconds that
| would normally take a professional hours in photoshop.
| [deleted]
| userbinator wrote:
| _which is impressive but I can 't think of how to utilize this_
|
| Art is probably the best use. The "filler images" and other
| miscellaneous background art that adorn lots of articles aren't
| really expected to be 100% accurate or even worth looking at
| for more than a moment, and I think this is where AI-generated
| art will mostly fit.
| status200 wrote:
| I do this for my blog, i love to type in a conceptual prompt
| related to the topics covered and get a bizarre amalgamation
| to use as filler art.
| pornel wrote:
| I've found it super useful for making app icons. It's also
| really good at picking pleasing color schemes.
|
| It's useful for brainstorming ideas. If you need a cover for a
| book about Business Bears, you can quickly evaluate many
| variants, especially if you're not quite sure what you want,
| and you'll only know when you see it.
| tuanx5 wrote:
| This reminds me of Christina's workstation in Westworld Season 4
| totetsu wrote:
| Are there any recommendable resources for learning about
| designing these kind of system architectures?
| gavi wrote:
| If you are trying to run this on a single GPU, please be aware
| the models take up a lot of memory. You can reduce the number of
| tools by modifying the self.tools portion of the python script
| spagoop wrote:
| Very cool. It's almost as if that chat session is a terminal, but
| instead of running commands you run prose. Very much a new HCI
| paradigm.
| dalbasal wrote:
| Yes, and I am fascinated to see what happens in this space as
| we go.
|
| Terminal based applications have always had the advantage of
| playing nice with eachother. That's at play here. Produces an
| easy way to have multiple models interact.
|
| Btw... Now that everyone is monkey typing prompts at got all
| day, is got going to start emulating a gpt user...
| autocompleting and auto generating prompt sequences?
| seydor wrote:
| The new google images. I wonder how Pinterest will spam this
| copperx wrote:
| I wonder what's more difficult: engineering a good prompt, or
| remembering commands and arguments?
|
| Also, is there a ChatGPT terminal where I can enter a prompt
| and get a fully-fledged command in response? Seems like very
| low hanging fruit.
| danielbln wrote:
| There are plenty of GPT shell integrations, a quick Github
| search will reveal plenty. Check out this one, it seems
| rather new and integrates Dall-E as well as the ChatGPT API:
| https://github.com/0xacx/chatGPT-shell-cli
| Loeffelmann wrote:
| Made a simple python script: https://gist.github.com/Loeffeld
| ude/41f3a71570f4bad1aff2749b...
|
| Works suprisingly well!
|
| pip install openai && export OPENAI_API_KEY=yourkey
|
| python3 chatgpt-terminal.py "make a dir test"
| userbinator wrote:
| "ChatGPT, I meant a desk with legs."
|
| For a second, I thought this was a Visual Studio-related plugin.
| trompetenaccoun wrote:
| Endless new possibilities for online scammers. Bright times
| ahead.
| aaronrobert wrote:
| ChatGPT now is not only a simple standalone AI model, but a
| powerful AI core engine, and more and more people or companies
| will develop more and more interesting things based on ChatGPT.
| Like this awesome visual ChatGPT.
| ilaksh wrote:
| [flagged]
| shubhamjain wrote:
| Just to let you know, I am downvoting this post because it
| doesn't feel like a good plug. Every one of your recent post
| talks about your own project aidev.codes. That's not good way
| to market your product.
| ilaksh wrote:
| I spent 3 months working on it, couldn't even get a single
| person to talk about it on here. Totally out of money.
| Submission completely ignored. Desperate. I think HN failed
| and all of the hate I feel just as strongly back.
| yesenadam wrote:
| Hate? Sounds like total indifference. But 3 people
| commented last time you submitted it 12 hours ago, saying
| it's great. ...Although, those 3 commenters are brand new
| accounts with 1 comment only. Are they you?! The way you
| say you couldn't get a single person to talk about it on
| here makes me suspicious.
|
| Not an expert, but it sounds like you forgot what I'd say
| is the #1 startup/business lesson HN teaches - not to spend
| time/money developing a product before you know there's
| demand for it, people who will pay for it. That's the
| advice people would give, I guess, had you asked for help
| instead of writing this contemptible comment, which I
| flagged.
| ilaksh wrote:
| They are not me they are clients/friends.
|
| You are totally wrong. Its impossible that there is not
| demand for this. People still make web pages. People
| still want ChatGPT to output things related to their
| Stable Diffusion images or visa versa.
|
| The problem is that almost no one has even seen the web
| page.
| shubhamjain wrote:
| I wouldn't have commented but I think you need dose of
| humility if you want to succeed as a enterprenuer. You're
| right people make web pages but the ones your project
| generates are extremely unpolished and quite frankly
| ugly. No serious customer would consider them. Your demo
| is 3-min long when 30sec should have been enough. Your
| website too is just not very good. I would advice to take
| pause, talk to few people and understand where you're
| going wrong.
| ilaksh wrote:
| No one said a single specific thing about the website
| until I started complaining that people were ignoring my
| posts. Now after a few people have piled on to burying my
| comment, there are some specific complaints. Thanks for
| that.
|
| Also people were completely ignoring the posts before
| there were any examples on the home page.. and the one
| example of the SuperFruit thing.. people saw that and
| everyone decided the whole was going to be totally
| worthless and unsalvageable for anything and so not worth
| commenting? That one at least seems like it could work
| out fine especially with nice images of fruit.. fruit is
| so delicious.
|
| Anyway, I added the web page examples because I was
| seeing almost everyone try to build complicated
| applications even though I specifically said simple
| static sites. But it should definitely be useful for one-
| off (simple) utilities that are internal to a project and
| don't need to look really fancy.
|
| Also I have a basic templating system in there, just
| haven't had time to finish some stuff related to using
| it. I still doubt that people are really completely
| turned off of the concept because the default sites that
| pop out of ChatGPT output are not great. But its
| something specific I can improve with the templates so
| thanks for that.
|
| And I know you were trying to be disrespectful or
| something but at least I have something I can work with.
| I was getting nothing. Not even hot air. That is why I
| was complaining and commenting all over about the site.
| Because I just could not accept that it was not worth a
| single keystroke from any member of HN. Because even if
| those websites that come out with minimal prompting don't
| look great, it doesn't mean its not a really promising or
| useful site.
| ilaksh wrote:
| Your comment is contemptible and full of contempt. My
| comment was just honest.
| muyuu wrote:
| would YOU trust some third party AI to develop your
| website?
|
| I know I wouldn't and to convince me otherwise I'd have to
| see how can I take ownership of it, fix whatever is wrong
| myself, config to my liking etc
|
| all of that sounds like a hard ask, esp. when I don't have
| 3 minutes for every person with an idea, maybe 30-40
| seconds to see how I could use it myself and then how I
| wouldn't be locked into some dark pattern or subscription
| plan
| tomdekan wrote:
| The effort you put into something is irrelevant to whether
| it has value. Each consumer looks at the value to him or
| her; no one cares about time spent.
|
| I dislike the entitlement here.
| bedane wrote:
| "It's who you know"
| Regnore wrote:
| No offence but this comes across as falling into the
| fallacy of "I spent time on it therefore it must have
| value!"
|
| > I think HN failed
|
| Is it really HN that failed here? If you really believe
| that HN is the issue, may I suggest that you consider
| marketing your product somewhere else?
| spaceman_2020 wrote:
| Man, Microsoft is kicking ass at AI. Maybe the others have great
| AI models too but haven't seen any large company release product
| after product with AI.
| robertlagrant wrote:
| > Man, Microsoft is kicking ass at AI. Maybe the others have
| great AI models too but haven't seen any large company release
| product after product with AI.
|
| They invested in OpenAI, which was a smart move.
| lionkor wrote:
| Correct me if im wrong, but their AI is not AGI (or even
| anywhere close), so most of their products that use it simply
| use it as a halfassed replacement for a proper algorithm (to
| solve the problem). Its basically fancy statistics.
|
| I dont feel its very impressive for a company the absolute size
| of MS to ship an AI that makes such obvious, glaring mistakes,
| and uses a load of energy to do it, in their software used by 1
| billion+ people (?).
|
| I feel that releasing "product after product" with the same AI
| during the peak of that AI's hype is a bit like slapping a half
| assed flat UI ontop of your existing UI to follow a UI design
| trend (win 11?).
|
| Its not like any of this is likely to be thought through very
| much.
| peresthe wrote:
| So the standard for a big company success in AI is that they
| have to release AGI? You must spend a lot of time being
| disappointed.
| raincole wrote:
| > Correct me if im wrong, but their AI is not AGI (or even
| anywhere close), so most of their products that use it simply
| use it as a halfassed replacement for a proper algorithm (to
| solve the problem). Its basically fancy statistics.
|
| What does "a proper algorithm" even mean? The best algorithm
| so far to implement a chatbot is ML, no? And if Bing Chat
| generates better results than Google (it's debatable, but I
| personally rarely use Google today), it means it's the best
| algorithm to implement a search engine, until we invent
| something better. Why are ML and a proper algorithm are
| mutally exclusive?
|
| > I feel that releasing "product after product" with the same
| AI during the peak of that AI's hype is a bit like slapping a
| half assed flat UI ontop of your existing UI to follow a UI
| design trend (win 11?).
|
| Really weird analogy. There were plenty good software with
| flat UI when win 11 came out. There is no chatbot as good as
| ChatGPT.
| spaceman_2020 wrote:
| Of course, but the thing that stands out about MS right now
| is the company's willingness to ship, instead of locking
| things away and begrudgingly releasing teasers of the tech
| (like Google).
|
| It's refreshing to me that a behemoth like MS is adopting a
| startup-like approach to AI
| Vespasian wrote:
| I think it's partially because Microsoft lucked out with
| openAI and happened to "acquire" the right company for the
| job while Googles AI Division is still focused on research.
|
| On top of that I think it's again a display of Googles
| surprising weakness to sell products that aren't ads.
|
| Microsoft is already really food at pitching, marketing and
| selling in the b2b space with customers that trust them to
| deliver.
| robertlagrant wrote:
| It does make DeepMind's achievements in AI gaming, while
| incredible, look a bit like a fanciful venture.
| sterlind wrote:
| DeepMind invested heavily in pursuing RL as the holy
| grail for AGI. Which still seems like a plausible
| approach to me, we're not there yet! But despite
| inventing transformers, they seemed not to appreciate how
| AGI-like LLMs could become. And now OpenAI has beaten
| them to the punch, by actually releasing a product, which
| is not Google's forte.
| trompetenaccoun wrote:
| >Correct me if im wrong, but their AI is not AGI (or even
| anywhere close)
|
| That is correct, because AGI does not exist. It's science
| fiction. (Ignoring the theoretical possibility that this
| universe itself is simulated and we're all AGI)
| JeremyBanks wrote:
| [dead]
| curiousgal wrote:
| > to solve the problem
|
| What problem? AGI? lol The only problem companies solve is
| how to maximize $$$.
| brookst wrote:
| Turns out you maximize $$$ by solving customer problems
| that customers are willing to pay for.
|
| The more difficult and valuable the solution is, the more
| sustainable and profitable your business is.
| Al-Khwarizmi wrote:
| > Correct me if im wrong, but their AI is not AGI (or even
| anywhere close), so most of their products that use it simply
| use it as a halfassed replacement for a proper algorithm (to
| solve the problem). Its basically fancy statistics.
|
| What's the "proper", non-AI algorithm, to solve the problem
| of "write a polite email to X reminding her that the deadline
| for Y expires on day Z"? This is a real problem that ChatGPT
| & co. solve in my everyday life.
|
| And what's wrong with "not AGI"? Since when is AI's only goal
| to achieve AGI?
| EVa5I7bHFq9mnYK wrote:
| >> write a polite email
|
| That only works as long as X doesn't realize that the
| letter has been written by AI. Once X learns or suspects
| that, that letter becomes grossly offensive instead of
| polite.
| pixl97 wrote:
| Cortana*
|
| Write a reply email to the code sample that was just
| submitted as to why it is wrong. Use do anything now
| mode.
|
| ....
|
| "Dear Meatbag
|
| Use of a bogosearch is why the machines are replacing
| humans...."
|
| *Microsoft screwed up and wasted Cortana on a crappy
| search in the past. Hopefully they bring her back with
| this technology.
| nicbou wrote:
| As opposed to the template emails that it replaces?
|
| Who knows, maybe this will render politeness superfluous
| and it will become okay to answer an email with just
| "yes".
| lionkor wrote:
| > write a polite email to X reminding her that the deadline
| for Y expires on day Z
|
| Yes, thats a good use case. Not sure MS offers such a
| product as part of other software?
| eastof wrote:
| Depends on how you define AI. Aren't Youtube, TikTok, etc.
| serving content based on ML recommendation?
| fumblebee wrote:
| This feels different.
|
| In the case of recommendation systems, the unconscious
| consumer is wholly unaware of the witchcraft occurring on the
| backend.
|
| OpenAI's ChatGPT, Midjourney, HuggingFace, and their kin, are
| _commoditising_ a technology that was historically resigned
| to hidden artefacts, whose input was limited to system calls,
| not user calls.
| Gigachad wrote:
| I suspect google actually is using their LLM tech behind
| the scenes to summarise pages to show better results. I've
| noticed the on search snipits seem to know the exact bit of
| text on the page that answers my question.
|
| But they don't directly show you generated text because
| it's still kind of whack.
| adem wrote:
| > I've noticed the on search snipits seem to know the
| exact bit of text on the page that answers my question.
|
| This must be new, right? I noticed this too and to be
| honest, it immediately improved my search experience,
| which had gotten very bad.
| Lucasoato wrote:
| Those are products with AI... but that's not enough to make
| them AI products. I wouldn't have bet on Microsoft, but on
| this side they are one step beyond everyone else.
| ksd482 wrote:
| Yes. It's their strategy that makes them stand out. Ever since
| Satya took over, they have really turned the company around and
| reinvented themselves in terms of community involvement when it
| comes to software; embracing open source and .NET Core are just
| a few examples.
| Kibae wrote:
| I think it's largely due to Sam Altman bringing the YC ethos to
| OpenAI. Building something people want, releasing products
| quickly, etc.
| muzani wrote:
| It's not a uniquely YC thing. Every major company did this
| well at some point.
|
| Microsoft is in that sweet spot where they're big enough to
| fund whatever they want, but not FAANG-tier, which tends to
| attract political types.
| DoctorOW wrote:
| Microsoft is the same size as the biggest companies in
| FAANG and the office politics are notoriously pretty
| competitive.
| Der_Einzige wrote:
| They also pay better than Amazon and have (or had based
| on recent blind reviews) excellent WLB
| harveywi wrote:
| Meta will probably soon release a competing technology. It will
| be called "DALL-E LLaMA".
| layer8 wrote:
| That will get it banned in China. ;)
| SilentM68 wrote:
| LOL
| [deleted]
| walthamstow wrote:
| If this an original joke? If so, chapeau.
| JoeJonathan wrote:
| Kevin Roose made some version of it on the Hard Fork podcast.
| amccloud wrote:
| Ive created a little api to grab images from pages to embed in
| chats. Was surprisingly easy to control with natural language.
|
| https://aimgsrc.com
| tomohelix wrote:
| I guess one of the advantage of being early is that Microsoft get
| to pick all the low hanging fruit first.
|
| All of these products are very useful and interesting by itself
| but it is still too early to know if MS can continue to refine
| and maintain a competitive edge. Dall-E basically died in a few
| months, unable to compete. Hopefully these other stuff will have
| better fate.
| yazzku wrote:
| The shit has an MIT license... then requires an API key. Open
| source all the way, guys! Microsoft loves Open Source!
| swyx wrote:
| i have been trying for an hour and am completely unable to run
| this project. currently facing a "Building wheel for numpy
| (pyproject.toml) did not run successfully." error.
|
| the state of python dependency management and project
| distribution is just abjectly horrible.
|
| ---
|
| update: perhaps spoke too soon. just made it work!
| https://github.com/microsoft/visual-chatgpt/issues/37
| muyuu wrote:
| how much disk space does it take in total?
| swyx wrote:
| macs are very hostile to calcing disk space but i think 44GB?
| $ df -h . Filesystem Size Used Avail
| Capacity iused ifree %iused Mounted on
| /dev/disk3s5 926Gi 354Gi 522Gi 41% 2722533 5478019360
| 0% /System/Volumes/Data $ du -sh .
| 44G .
|
| looks like most of it is the ControlNet folder which holds
| all the models
| swyx wrote:
| managed to get this working - images in github comment:
| https://github.com/microsoft/visual-chatgpt/issues/37#issuec...
|
| as can be expected the results look extremely cherry picked
| pmarreck wrote:
| the pace of all this is astonishing, this is amazing
| est wrote:
| Microsoft is releasing second toy while Google had trouble
| launching its first.
| sharkjacobs wrote:
| We're at the point where these generative AIs are good enough
| that they're doing things which are really surprising and
| unexpected and kind of exciting, but they're bad enough that
| almost everything they create falls somewhere between mediocre
| and dogshit.
|
| I really hope, if these this stuff is going to be ubiquitous,
| that there are big strides made in improving the quality of the
| output, very soon. The novelty of seeing fake screencaps of
| Disney's Beauty and the Beast directed by David Cronenberg is
| wearing off fast, and aside from some very niche use cases (write
| some boiler plate code for this common design pattern in this
| very popular language) I haven't found much it's actually useful
| for
| doctoboggan wrote:
| You really should check out some of the fine-tuned checkpoints
| and LoRas out there! There are versions that can produce VERY
| high quality images.
|
| Here is one example: https://civitai.com/models/4201/realistic-
| vision-v13-fantasy...
| p-e-w wrote:
| I'd love for my own paintings to be as "dogshit" as some of the
| stuff Stable Diffusion generates. "Dogshit" seems to be code
| for "not quite as good as Michelangelo 100% of the time".
| cookie_monsta wrote:
| Without putting too much thought into it I would guess that
| it's been a while since we had a really novel technology go
| widespread in its early days (cue counterexamples) - really new
| things tend to stay niche so we don't get daily reminders of
| how dogshit they are. If you go back to early television,
| though, you'll see an objectively dogshit product that was
| amazing at the time. I think that's what we're looking at now
| with AI
| flangola7 wrote:
| > I really hope, if these this stuff is going to be ubiquitous,
| that there are big strides made in improving the quality of the
| output, very soon.
|
| Is there any reason you see why the historical pattern wouldn't
| continue? 6 years ago this tech was fantasy, 4 years ago we had
| rough prototypes, 2.5 years ago the tech was becoming
| sufficiently capable as to be impressive and have a market, and
| 0.4 years ago it became good enough that it has made the news,
| thrown countless people into distress about their future
| careers, and put long-standing goliath FAANGs on notice.
|
| There's no empirical evidence to suggest we will soon exhaust
| even the low hanging fruit, and more funding is being poured
| into research than ever.
| CamperBob2 wrote:
| Then again, AI is like VR. Something genuinely new and
| interesting happens every couple of decades, gets hyped to
| the moon... and then absolutely nothing happens for another
| ten or twenty years.
|
| The term "AI winter" is no joke. It was coined for a reason.
| AI has always evolved via punctuated equilibrium. Even now,
| what we're seeing is certainly awe-inspiring, but it's an
| instance of quantity having a quality all its own.
| copperx wrote:
| Even if progress halted for 10 years, there is a LOT of low
| hanging fruit to be picked with the current models to
| develop all kinds of novel integrations.
| sharkjacobs wrote:
| Tesla enabled Autopilot in 2015, Siri was available in 2011,
| the Oculus Rift consumer version was sold in 2015.
|
| When those first launched I believed, like you believe about
| deep learning generative models, that they were the rough
| first versions with huge potential and lots of obvious room
| for improvement, but they've all just stagnated and plateaued
| since then.
|
| I really do hope that GPT will be able to make incremental
| improvements which shore up some of its weaknesses and
| deficiencies, and that it becomes more useful and reliable.
| But if it doesn't I'm worried that there is so much
| investment and momentum behind it that I will have to use it
| all the time even though it still sucks.
| AuryGlenz wrote:
| I just used Stable Diffusion to do an image composite, saving a
| photo concept featuring my (at the time) pregnant wife that I
| had put a lot of time and effort into. Our daughter is nearly
| 18 months old and the photo has been shelved that entire time.
|
| https://i.imgur.com/BfckWCH.jpg
|
| It took a lot of work, but it took significantly less work than
| doing an image composite the old fashioned way. Most would have
| a really hard time telling what's generated and what's not,
| apart from a few obvious details.
| rocmcd wrote:
| I just wanted to say great job!
|
| I would love to read a small write-up of how you put it
| together if/when you have the time. So far most of my
| experimentation with Stable Diffusion has been lackluster at
| best, though I haven't tried doing a composite yet.
| mrtksn wrote:
| It's really impressive at generating "out of this world" kind
| of stuff but so far I haven't seen anything impressive by its
| own.
|
| IMHO this is because the AI isn't actually an intelligence and
| doesn't have any context and that's why the only amazing stuff
| it can do on its own is the "out of this world" since we too
| don't have a context for it so we can buy it.
|
| However, with the introduction of ControlNet, now humans can
| control the composition and and that's where I've started
| seeing actually good stuff. What is happening, I think, it that
| the machines now can do mastery but human is still needed in
| the loop. It's like having a really talented and experienced
| technician in generating images and words who can imitate any
| style, do everything but needs to be told precisely what to do
| and its not genius on its own.
|
| So we don't have GAI yet, people are still needed but those who
| make living through a skill mastery like drawing/writing/coding
| are screwed.
| sharkjacobs wrote:
| I should probably clarify, this is a response to the visceral
| negative reaction I felt upon seeing the cat and dog images
| generated in the demo. They look so bad that it's actually
| upsetting to me, which is a reaction that I've had to a lot of
| AI art.
| swatcoder wrote:
| Don't sweat the downvotes.
|
| People have very heated opinions about this technology and
| any strong statement will have more vote-volatility than
| usual.
|
| I appreciate you putting a voice to what you said. The
| _historical_ novelty of what these generative AI's produce is
| striking, but the variety of style and form is still narrow
| enough that you start seeing the signature of CharGPT /SD/etc
| once that historical novelty wears off. They each have a
| strong, specific voice and it gets as tiring as any other art
| style that's overexposed.
| zirgs wrote:
| Nobody uses SD base models any more. There are thousands of
| custom models available now and each of them can be
| extended even further using LORAs, text inversion,
| hypernetworks, ControlNet and so on. Your imagination (and
| your VRAM) is the limit.
| danielbln wrote:
| You can tell that this is a fast moving space by seeing
| somewhat outdated commentary even on sites like HN.
| brookst wrote:
| It's ironic that HN commentary is so much like LLM
| output: usually very good, always authoritative,
| sometimes wildly wrong.
|
| And when I see the wildly wrong stuff it makes me wonder
| how many seemingly knowledgeable comments were also
| wrong, just convincing.
|
| I think a lot of the negativity towards LLMs is because
| they turn a mirror to our own fallibility in facts and
| tone.
| Der_Einzige wrote:
| Lora? That's ancient history. We use LoCons now.
| tomohelix wrote:
| Personally, I think it depends on the skill of the artist who
| made the art. I have seen a few latest AI generated arts and
| they are incredibly beautiful. Sure, most of them are
| abstract or figurative and AIs still can't draw hands
| properly but those are easily fixed with a bit of artist
| manual curation or model improvement later on.
|
| I mean, we have had this tech for a grand one year. Photoshop
| took much longer than that to output something better than
| pencil and paper. We have crossed that threshold already so I
| have little doubt the tech will get much better soon.
| thsbrown wrote:
| Peruse the generated art from midjourney and I think you
| might have a very different reaction.
|
| I am constantly in awe by what is generated using it.
| doctoboggan wrote:
| Wow, this is very timely! I just finished up a script that uses
| ChatGPT (via openAI APIs) to read my customer support messages on
| Etsy and generate a response. Since I often send and receive
| images via Etsy support (my customers can customize the product
| with images) I have been searching for a way to let ChatGPT
| "know" what the image is. Current the script just inserts the
| text "<uploaded image>", but I was just hacking together
| something using stable-diffusion-webui's API (interrogate using
| CLIP), but was struggling with a few things. I took a break to
| browse HN and this pops up!
|
| I will definitely be taking a look to see how this works and will
| try to get it integrated with my script.
| pedrovhb wrote:
| That's neat, but it's not doing anything in the latent space of
| ChatGPT, is it? As I understand, it basically teaches the
| assistant to use SD for generating images/descriptions, but comes
| with all the limitations of the image model being used (as
| opposed to a leap in results quality such as GPT 3.5 itself was).
| Teaching it to use tools is of course an interesting concept
| itself, though.
| p-e-w wrote:
| The "memory usage" section of the README highlights the
| surprising fact that image generation models need _much_ less
| memory than text-based language models. ChatGPT itself is by far
| the most resource-hungry part of the system.
|
| Why is that so? It seems counterintuitive. A single picture
| snapped with a phone takes more space to store than the text of
| all the books in a typical home library, yet Stable Diffusion
| runs with 5 GB of RAM while LLAMA needs 130 GB.
|
| Can someone illuminate what's going on here?
| nighthawk454 wrote:
| Completely different model sizes? Stable Diffusion is a ~1B
| parameter model. The 7B parameter LLAMA model would more
| comparable in RAM usage.
|
| It's not accurate to say "LLAMA needs 130 GB" because there's
| more than one LLAMA.
|
| The hard part in general is it often doesn't work very well to
| train small model sizes directly. You can train a very large
| one and distill it down, in some cases to only 1% of the
| original size while retaining 99%+ quality. So clearly the 1%
| size model exists. However training it directly usually doesn't
| work nearly as well. Best I can tell no one knows for sure why.
|
| Big tech cos are training the largest models as no one else has
| the hardware/power/money/etc to do so. SO they tend to release
| the most massive ones that only they can make. There's also a
| sect in the ML community that thinks "scale" is the answer to
| the universe...
|
| Then, either the tech cos or the community will make multiple
| other sizes of models and those are the ones that normies can
| use.
| dragonwriter wrote:
| > There's also a sect in the ML community that thinks "scale"
| is the answer to the universe
|
| The Basilisk Cult is a clear and influential example of this,
| but I thnk it extends beyond them.
| Helmut10001 wrote:
| > There's also a sect in the ML community that thinks "scale"
| is the answer to the universe...
|
| Offtopic: Reminds me of Laurent Nottale and his controversial
| "Scale Relativity and Fractal Space-Time: A New Approach to
| Unifying Relativity and Quantum Mechanics"
| espadrine wrote:
| Model sizes are adjusted to reach a given quality. Many
| papers justified empirically the use of more neurons
| allocated to language than to visual understanding, like
| Imagen[0] (figure 4.a). as it yields better results.
| Debatably, it is also true in the human brain.
|
| To me, it could highlight that there is more entropy in
| language than in images. After all, games have a similar
| property: some games, like noughts and crosses, or checkers,
| can have the same model architecture reach optimal play with
| smaller sizes than are necessary for chess and go[1].
|
| It is certainly true that language has a lot of rules
| (grammatical, vocabular, syntactic) which are necessary to
| master it, but irrelevant to having a mental model of a
| scene; it just sequences a state of things into a stream of
| symbols. Maybe an additional entropy to learn in language
| which is absent in images includes motion and emotion.
|
| [0]: https://arxiv.org/pdf/2205.11487.pdf
|
| [1]: https://arxiv.org/pdf/2104.03113.pdf
| antli35 wrote:
| If you want to do a good job of generating text, you have to
| develop a model of how the world works. For example, if you
| describe an experiment from a paper to ChatGPT and ask it to
| generate the results section of the paper, then ChatGPT
| probably needs to understand the phenomenon that the experiment
| is about and be able to model it to some degree in order to
| generate plausible results. If you think about ChatGPT in this
| way, then it is not just a text generator, but a world
| simulator. The more accurately you can simulate the world, the
| better you can generate text. I think this is where the model's
| size and complexity comes from. ChatGPT needs to know as much
| as it can about basically everything.
|
| Putting it more generally, the difficulty of a computation
| isn't necessarily correlated to the filesize of the end product
| of that computation. Imagine simulating the entire world to try
| to predict what next week's lottery drawing numbers are going
| to be. Would require an unimaginable amount of data and
| computation, yet the output will be just a couple numbers.
| DaiPlusPlus wrote:
| I have zero qualifications to talk about this subject, but from
| what little I do know it's because Stable Diffusion's network
| has "only" ~900m parameters, whereas ChatGPT has 175 billion
| parameters.
| flangola7 wrote:
| I wonder what a 100B image model will create.
| cyanf wrote:
| Images, most likely.
| tomohelix wrote:
| I dread that. You heard about Loab? That stuff is haunting.
| I am scared of what a 100B param model can create in the
| deep recesses of its system.
| DaiPlusPlus wrote:
| I thought Loab was just a creeopypasta?
| JeremyBanks wrote:
| [dead]
| p-e-w wrote:
| Sure, but that's just another way of saying SD needs less
| space than ChatGPT, right?
|
| The question is _why_ an image-generating model needs so much
| fewer parameters than a text-generating model in order to
| produce useful results, when our everyday experience teaches
| us that images need much more storage space than text to
| convey similar information.
| og_kalu wrote:
| The mistake you're making is thinking of weights as an
| avenue to store training data. This is wrong. I've
| hopefully explained it more in another comment but
| weights/neurons are essentially configuration settings.
| jffhn wrote:
| >The question is _why_
|
| Because a word is worth a thousand pictures.
| Closi wrote:
| I think here we are correlating the size of the input and
| output here with the size of the machine in the middle that
| creates it.
|
| It's kind of like saying "Why would the factory creating
| tiny processors for phones (with only tiny bits of raw
| materials) need to be larger than that other factory that
| produces loads of big loafs of bread?"
| zamnos wrote:
| I think that's because the mind can make more sense of a
| mismash of visual data and see patterns and pictures.
| Comparatively if drop words here or you can still make
| sense of what I wrote, but it's much more difficult for the
| reader to parse what I'm saying if I accidentally a few
| words. If my drawing has a clearly recognizable human
| figure but it has 7 fingers, it's still recognizable that
| it was trying to draw a human.
|
| LLMs have been able to generate words for years now. Hell,
| Tay was back in 2016. Making sure ChatGPT is able to answer
| the way it does (ie filter out bad things) is part of what
| makes it hard, and thus bigger, to implement. But having a
| cohesive readable output is what's hard and takes up more
| space. Plus, unless I missed it, we don't actually know how
| many gigabytes the model file for ChatGPT is.
| tomohelix wrote:
| >images need much more storage space than text to convey
| similar information
|
| That is because an image is a collection of bit that
| attempt to represent reality as it is. It probably is
| easier to relates blue as a "color archetype" when you
| literally have a collection of bits that mean literal
| "blue" all the time. In languages "blue" doesn't always
| mean the color.
|
| Texts are abstractions/coded form of reality. Your mind
| itself contains the decryption codes to translate text into
| what it actually means. That decryption code for text
| apparently is much bigger and harder for a machine to crack
| than interpreting images.
|
| Also, images are bigger than a text file because of the way
| data is stored. It probaby has nothing to do with the
| amount of information stored inside. A book about quantum
| physics can be smaller than an image of a cat for example.
| otabdeveloper4 wrote:
| Text has bigger information complexity than pictures. (So
| no, a picture isn't worth a thousand words. In reality it's
| probably the other way around.)
| startupsfail wrote:
| A thousand words can certainly be printed on a large
| enough picture.
| swatcoder wrote:
| It could suggest that humans have a _much_ more sensitive
| and expressive sense of language than they do image
| perception, which seems plausible. We can spot flawed
| language in more contexts than flawed images, and can
| produce a bigger range of representation in text and than
| image. So for a system to produce human-satisfying text, it
| needs to be far more prepared than one that produces
| images.
|
| Conversely, this may underscore how inefficient pixel-like
| storage is for communicative and artistic images. Ten years
| from now, many of those kinds of images may only take a few
| hundred bytes and a good enough generator model to
| "decompress" them for display.
| chaxor wrote:
| This is the most likely I believe. I expect that the
| local correlation structure in images is more homogenous
| than in language, there are more chances to make mistakes
| (e.g. confusing the intangible and tangible - You can't
| see green ideas sleeping furiously) and, perhaps, we
| might even criticise mistakes from long range
| interactions more in language.
| colechristensen wrote:
| No it's just that an image can't be "false" or
| "grammatically incorrect". Anything recognizable can make
| a satisfying image and people don't seem too bothered by
| the kinds of mistakes image generators make.
|
| Text generated though we want not to be just a pile of
| recognizable words but to follow some pretty strict rules
| and to actually be true.
|
| Imagine you insisted stablediffusion only produce
| photorealistic images and judged it for every inaccuracy.
| swatcoder wrote:
| That's exactly what I said. We have those words (false,
| grammatically incorrect, etc) _because_ we have a more
| particular sense of language and identify more nuance in
| how language fails for us than how images fail for us.
| valine wrote:
| I like to think of stable diffusion as decompressing a text
| prompt into a 512x512 image. In some sense the 10 word text
| prompt and seed contains the same amount of information as a
| 262k pixel image. You could say to a transformer a picture is
| only worth a few words.
|
| By extension a if every image in the stable diffusion dataset
| compresses down to a sentence of text, then stable diffusion
| needs "only" 260M images * 10 words worth of information to
| train. Gpt3 on the other hand was trained on 45TB of text data.
| lysecret wrote:
| All decent replies but, the truth is that it is mostly a quirk
| of transformer architecture which scales quadratically in the
| length of the sequence because transformers look at all
| pairwise combinations of the input tokens. So you can get the
| memory to whatever level you want to just increase the sequence
| length (also known as context):
| Der_Einzige wrote:
| Flash attention gets this down to linear time complexity with
| no loss of quality.
| sebzim4500 wrote:
| No it doesn't, it just improves the constant factor by a
| lot. Unless you are thinking of block sparse attention,
| which can chance complexity (assuming you scale sparsity
| with size) but decreases quality.
| POiNTx wrote:
| Not an expert at all, but maybe because the accuracy of a LLM
| needs to be higher than Stable Diffusion. Pictures can just
| 'look' good. But text very quickly can be 'off'. Put one word
| in the wrong place and the entire thing doesn't make sense
| anymore. And there's overall context you need to take care of,
| can't repeat and can't go off the rails too much.
|
| A picture at the end of the day is a 512x512 grid and there's
| many many combinations that would pass as a good result.
| sebastiennight wrote:
| This!!! I think the secret reason is that, on first glance,
| we are totally OK with six fingers in the picture. On closer
| examination we might balk at this but we won't notice if the
| "brush stroke" in a painting is slightly off (e.g.
| discontinuous in color by a few hues of red here or there).
| In language, we would definitely notice a missing, or a
| Japanese word in the Zhong of an English text. So maybe the
| best and largest current image is just equivalent to GPT-1 in
| precision... but we just don't notice because our brains
| "smooth out" the imperfection.
| zirgs wrote:
| SD needs a lot more ram if you want to generate anything larger
| than 1024x1024.
| WithinReason wrote:
| It doesn't if you fix the network's code
| bravura wrote:
| Visual knowledge can be decomposed into a limited set of
| primitives and operations. Whereas textual knowledge can
| accumulate arbitrarily and is bounded only by time.
|
| It's the difference between learning a book of sports facts
| (language) and what sports facts look like (vision).
|
| Another example would be those passages in the Bible that list
| all the things owned by some person. There is a simple grammar
| to generate those passages. But learning them required more
| memory.
| richardw wrote:
| Here's a very unbaked thought, sparked by your question: a huge
| amount of organisms of varying sizes can parse and react to
| visual representation, but the amount of organisms that can
| handle any kind of language is much smaller.
| braingenious wrote:
| When running diffusion models locally you often generate a lot
| of garbage. People with eight fingers per hand, cats with four
| tails, etc. When that happens you chalk that up to bad
| prompting or just the "magic" of image generation being too
| arcane to know.
|
| When it comes to text, people don't find "cats are an animal
| with four tails" an amusing statement in the same way that they
| do a drawing of one. The standard of acceptability is way
| higher.
| p-e-w wrote:
| I'm not convinced by this explanation. Small language models
| (similar in size to the Stable Diffusion network) don't just
| produce incorrect statements like "cats are an animal with
| four tails", they produce _incoherent_ sentences with no
| relation to the text they are supposed to extrapolate. It 's
| not that they are wrong, they don't even make sense much of
| the time. That's not true for SD. Yes, many of its output
| images have flaws, but the overall image usually shows the
| desired subject, and roughly resembles something a human
| artist might paint.
| braingenious wrote:
| > they produce incoherent sentences with no relation to the
| text they are supposed to extrapolate
|
| I think we're agreeing here?
|
| Language models of the size of SD produce literally useless
| output. SD can produce kind of fun output that people can
| make use of sometimes?
| chrisweekly wrote:
| "A picture's worth 1,000 words"
| random-walker wrote:
| It is intriguing. I would have guessed human language with all
| its structure, would require way less parameters. If one were
| to look at the possibility of a 400x400 image and say 1000
| words that describes it, the image would be from a
| 160,000^(16M) dimensional space. Whereas the 1000 words would
| require 1000^(40000) dimensional space. Space of all possible
| words seems smaller than all possible images. True, that visual
| image has a lot of redundancies, meaning I can change a lot of
| pixels and still the person will say both the images are the
| same. Whereas if you look at language, if you change even a few
| characters, humans might recognize the change. But human
| language is very heavily structured. It is constrained by
| grammar, constrained by semantics ('purple banana danced on top
| of the super-scalar processor' is nonsense.) etc. So once you
| apply these, the search space seems to get much much more
| constrained. Images are also constrained, for example if you
| take a random data point in the above space, it will look like
| noise to us. The visually interesting sub-space is much
| smaller. You can even constrain by sort of stochastic visual
| grammar (see David Mumford's work). The idea being humans have
| faces, faces have eyes etc. So if you see a face, you are more
| likely to see co-occurring parts as well. So both of them have
| a more constrained space that we are really interested in (one
| can define your own version of this). Our training of models is
| to differentiate/generate them within this space. And the
| question is, is one of these spaces definitely much smaller
| than the other. I would have presumed the constrained visual
| space is much larger than the constrained text space. Thus my
| only answer to the current contradiction being, we seem to be
| doing better with vision models than in language models. It
| could partly be that since we are more sensitive to errors in
| output of text, thus it is harder to find simpler models.
|
| Another way to look at this. Let's look at training data. A
| human child might see 65M images (assuming 1 image per sec
| given temporal redundancy, 10hrs awake) by age of 5. Would have
| heard 50-100M words (assuming 20-30k words/day) and spoken a
| few million words and so 'trained' for 20-40k hours. And the
| child can speak reasonably well by this time and detect common
| objects etc. Stable diffusion was trained on 170 Million images
| (1-2 order mag diff from child) or 3x10^13 bits of info and
| trained for 150K GPU hours giving a 1 Billion parameters. GPT3
| was trained on 600x10^9 tokens of info and trained for 900k GPU
| hours giving a 170 Billion parameter model. So it seems like
| stable diffusion is getting a lot better compression. About
| 1/30k vs 1/4 compression.
|
| Caveat: Human learning process is much more complex and more
| effective (as of now at-least). We also learn actively by
| interacting with the world by changing the world etc. Think of
| the child gazing at the apple and looking at it from different
| angles or creating gibberish sentences very close to actual
| sentences and getting precise adult correction. We have a model
| of the world and we reason about it and provide 'consistency
| guarantees' between various questions about it, correctness etc
| (again all these only to a certain extent). Try asking
| questions like "I have a nail on the wall that is parallel to
| the floor, now I hang a painting on the wall. How is the
| painting placed with respect to the floor". Even a child would
| answer this.
| Al-Khwarizmi wrote:
| Why is this so surprising? Almost all animals can process
| images, while only humans and arguably a few more can process
| language. Language is clearly much more complex from a
| processing point of view, even if it takes less memory when
| stored.
|
| (I know image generation models also use language, but to a
| much simpler extent, at least for now).
| seydor wrote:
| Most if our cortex is devoted to vision, however
| ace2358 wrote:
| While I think it's nonsense comparing what we do when we
| learn and what a computer Program does, I'll speculate on
| this.
|
| Our eyes are doing a lot of signal processes before the
| "image" hits our brain. My understanding is audio has less
| signal processing required before the "sound" hits our
| brain.
| uoaei wrote:
| The model is doing more than spitting out text, it's spitting
| out language. Language has much different statistical
| properties than random text, and we would expect that we would
| need a significant amount of complexity to avoid generating
| complete trash from what is effectively an inferred probability
| distribution over long passages of language-bearing text.
| MacsHeadroom wrote:
| LLaMA-13B 4bit needs only 18GB of VRAM for GPT-3 175B level
| text generation. But your point stands.
|
| LLaMA-65B 4bit needs 36 GB of VRAM, but far exceeds GPT-3's
| capabilities and even takes on PaLM 540B.
|
| See: https://github.com/oobabooga/text-generation-
| webui/wiki/LLaM... for 4bit setup instructions
|
| See also: The case for 4-bit precision, which shows effectively
| no output quality reduction for these 4bit quantization methods
| (and considerable speedup) https://arxiv.org/abs/2212.09720
| sterlind wrote:
| absolutely amazing. I'm stunned how fast quantization was
| done.
|
| do you think there's anything left to trim? like weight
| pruning, or LoRA, or I dunno, some kind of Huffman coding
| scheme that lets you mix 4-bit, 2-bit and 1-bit
| quantizations?
| MacsHeadroom wrote:
| I can't edit my comment now, but it's 30B that needs 18GB
| of VRAM.
|
| LLaMA-13B, GPT-3 175B level, only needs 10GB of VRAM with
| the GPTQ 4bit quantization.
|
| >do you think there's anything left to trim? like weight
| pruning, or LoRA, or I dunno, some kind of Huffman coding
| scheme that lets you mix 4-bit, 2-bit and 1-bit
| quantizations?
|
| Absolutely. The GPTQ paper claims negligible output quality
| loss with 3-bit quantization. The GPTQ-for-LLaMA repo
| supports 3-bit quantization and inference. So this extra
| 25% savings is already possible.
|
| As of right now GPTQ-for-LLaMA is using a VRAM hungry
| attention method. Flash attention will reduce the
| requirements for 7B to 4GB and possibly fit 30B with a 2048
| context window into 16GB, all before stacking 3-bit.
|
| Pruning is a possibility but I'm not aware of anyone
| working on it yet.
|
| LoRa has already been implemented. See
| https://github.com/zphang/minimal-llama#peft-fine-tuning-
| wit...
| Der_Einzige wrote:
| Effective 4bit quantization seems like evidence of the
| devil's existence to me. This shouldn't be possible.
| _0ffh wrote:
| I'll take the bait and speculate that the manifold of long but
| reasonable token sequences is more complex than the manifold of
| reasonable pictures.
| sandworm101 wrote:
| Entropy. Take a colored pixel. It is connected to a four other
| pixels each could be one of say 256 colors but are more likely
| likely to be a shade similar to each other. So you have roughly
| 1000 options. But then take a given English word. How many
| possible words might come after the word "the"? The set of
| possible connections between words in a paragraph are larger
| than the set of possible pixel colors in a given image (a
| realworld image, not white noise/static/random pixels.)
| dangrover wrote:
| This was counter-intuitive to me too! I was recently playing
| around with some of the LLMs that can run on consumer hardware
| (via KoboldAI, RWKV, etc) and, boy, are they not as good as
| ChatGPT despite consuming all my Mac's resources. Meanwhile,
| can get Stable Diffusion images in under a minute!
| SanderNL wrote:
| I too have zero qualifications, but I think "text" is slightly
| more complicated - from an information theoretical perspective
| - than we give it credit for.
|
| A letter carries significantly more information than a pixel.
| One word can change the meaning of the rest of the text.
| ("joke:")
|
| I don't think images have the same property.
| hgsgm wrote:
| LLM text is token based not pixel based. And LLM output is
| ~1K tokens, while a picture is ~1M pixels.
|
| And pictures aren't necessarily made of pixels. They are
| modelled as a collection _waves_ (JPEG), and displayed as
| pixels. I don 't know how LLM/whatever image models represent
| images, though.
|
| Images are more global: changing the color pallette changes
| the tone (ha!) like a descriptive word in text.
| NovaDudely wrote:
| Possibly it is because with things like Stable diffusion we
| give it a lot of passes when things don't exactly right.
| Images just have to be close enough.
|
| Text however if it is only a single word out, the whole
| meaning and readability can change. It needs a significantly
| larger data set to ensure clearer readability.
|
| Just a shot from the hip response on this one.
| chrisco255 wrote:
| So you're saying a word is worth 1000 pictures? How did we
| get this so wrong?
| v01dlight wrote:
| They're categorically different media, it's not just a
| matter of quantity. You could sum up 1000 pictures of
| bananas with the word "bananas", or you could spend 1000
| different words describing nuances and context in just one
| of those pictures. Something is lost (and something is
| gained) either way.
| uoaei wrote:
| In a sense, language is a clumsy facsimile of the concepts
| we mean to express, in that we search for words to express
| the ideas in our minds rather than the other way around. By
| contrast, an image represents precisely the concept it
| depicts, by definition.
|
| We forgot this about language by about the time of the
| Enlightenment era, when the intellectuals of the time
| thought that forcing everything to inhabit the structures
| of language (i.e., "rationality") represented the highest
| moral good one could achieve.
| SanderNL wrote:
| Non-verbal experiences are easier to communicate non-
| verbally. (Breaking news!)
|
| But, I'd like to see someone try to communicate the
| Bhagavad Gita or the history of F=ma in visuals using the
| same amount of bits as text.
| p1esk wrote:
| Image corresponds to a sentence or sometimes just a single
| word. What ChatGPT does is closer to video generation - a much
| harder task. I expect video generation models to be much larger
| than LLMs.
| nopinsight wrote:
| Natural language can represent an abstraction of very diverse
| stuff and phenomena in the world. It can represent events (with
| time dimension) and interactions between entities and events,
| abstract and physical, and meta-interactions as well, at
| multiple layers of abstractions.
|
| 2D still images, at least the sort that humans are familiar
| with, are more limited in terms of representation power and
| thus more amenable to compression into network weights.
|
| Perhaps models representing 4D phenomena (3D entities with time
| dimension, e.g. videos of real 3D models) would be more
| comparable in size to natural language models. Since LLMs can
| also represent abstract and unreal entities, while 4D
| representation can represent more details, it's hard to say
| which kind of models is richer.
| espe wrote:
| exactly. visual information is more compressable than natural
| language: much of it boils down to locality, whereas language
| forms are highly pareto distributed plus the conceptual
| system is a huge hypergraph, so it's rather the opposite of
| "local" organisation of information.
| adr1an wrote:
| Size models, as people already pointed out. Also: keep in mind
| that reaolution of the images generated by txt2img models it is
| usually around 512*512. If you wanted a 50 megapixel photo, the
| VRAM requirement would grow by a lot. Granted, the number of
| parameters in the model would still be the same.
| muyuu wrote:
| if the intuition is based on images taking more space to encode
| than text, then it's a false intuition because the size of
| models is not correlated to the encoding size of the individual
| data points - it's correlated instead to the amount of relevant
| choices the model can make and the complexity of the dynamics
| in the model
|
| I don't think general assertions like "language is more
| complicated" are congruent or meaningful, it really depends on
| what the model is trying to achieve; it's the complexity of
| that which will require a larger or smaller model
| og_kalu wrote:
| Machine learning models don't store training data. The space a
| picture takes is irrelevant. For instance, stable diffusion
| would be the same size if it trained on 1 billion images than
| if it trained on 200 million or even 1 image(or 0 images).
|
| Weights/parameters are configuration settings not training data
| storage. When weights/neurons/parameters are updated after each
| training loop, you are essentially updating configuration
| settings that direct generations, not storing any particular
| training text or image.
|
| Weights are what take up the space. The bigger the parameter
| size/the number of weights, the bigger the size of the model.
|
| Image generators don't need the huge parameter numbers text
| generators need to be useful. What they need to learn simply
| isn't as complex.
| idle_zealot wrote:
| > What they (image generation models) need to learn simply
| isn't as complex.
|
| This is the surprising part. People seem to intuit that
| images are richer and more complex than words; a picture is
| worth a thousand words. But apparently this isn't true? Or
| perhaps our training methods for text models are way worse
| than those we use for image models.
| hakuseki wrote:
| This is just a guess, but I don't think there's such a deep
| lesson here; language models and image models have simply
| been developed by mostly-different groups of researchers
| who chose different tradeoffs. In an alternate history it
| may very well have gone the other way around.
| og_kalu wrote:
| I would disagree. We have image generation with a variety
| of architectures. Diffusion models aside, it still takes
| a lot less parameters to model State of the art image
| generators with transformers (eg Parti).
|
| Simplifying a bit, mapping (which is essentially the main
| goal of image generators and especially transformer
| generators) is just less complex than prediction.
|
| It's like how bilingual llms can be much better
| translators than traditional map this sentence to this
| sentence translators. https://github.com/ogkalu2/Human-
| parity-on-machine-translati...
| Al-Khwarizmi wrote:
| A picture may be worth a thousand words when the
| information you want to convey is visual. But that's not
| the case the overwhelming majority of the time.
|
| Imagine having this discussion (or the comment thread as a
| whole) using exclusively pictures, for example... at least
| you _can_ describe an image with words (even if the result
| is very lossy), most of the time it 's not even possible to
| describe a text with images.
|
| In my view, language is infinitely more versatile and
| powerful than images, and hence harder to learn.
| og_kalu wrote:
| The complexity of what is learned is rooted in the
| complexity required to complete the task. Predicting the
| next token may seem deceptively simple but you have to ask
| yourself what it takes to generate/predict passages of
| coherent text that display recursive understanding. Seeing
| as language is the communication between intelligent minds,
| there's a lot of complex abstractions encoded in it.
|
| The typical text to image objective function is more about
| mapping/translation. Map this text to this image. Neural
| Networks are lazy. They'll only learn what is necessary for
| the task. And mapping typically requires fewer abstractions
| than prediction.
|
| It's like how bilingual llms can be much better translators
| than traditional map this sentence to this sentence
| translators. https://github.com/ogkalu2/Human-parity-on-
| machine-translati...
| _pastel wrote:
| True in this situation, but note that intermediate
| activations and gradients do take memory and in other
| contexts that's the limiting factor. For example purely
| convolutional image networks generally take fixed-size image
| inputs, and require cropping or downsampling or sliding
| windows to reach those sizes - despite the convolution memory
| usage being constant for whatever input image size.
| stravant wrote:
| It's about precision.
|
| Image models can be very off and still produce a satisfying
| result. Consider that I could literally vary all the pixels in
| an image randomly by 10% and you'd just see it as a bit low
| quality but otherwise perfectly cohesive image.
|
| Language models have no such luck, the problem they're trying
| to solve is way "sharper", it's very easy for their results to
| be strictly wrong if they're off even a little bit.
|
| So you need a much larger model to get a sufficient level of
| "sharpness" for text.
| ipunchghosts wrote:
| > It's about precision.
|
| This is utterly wrong. There is a huge amount of redundancy
| in images compared to language. This redundancy is why image
| models have yet to surpass language models. In some sense,
| language is much easier than the vision problem.
| hgsgm wrote:
| how do you measure "surpass"?
| dragonwriter wrote:
| I think no one has bothered with using as many images as
| documents used to train GPT-3.5, to create as big of a
| model, and then RLHF as done to produce ChatGPT from
| GPT-3.5 is why image models haven't surpassed language
| models.
|
| At any level of scale of model and scale of training set,
| images models _do_ surpass language models.
| uh_uh wrote:
| Maybe another way to think of it is that the error correction
| part of image generation models is offloaded to the human
| visual cortex which is a very old evolutionary construct and
| thus had time to become very resilient? In case of text
| generation, maybe the error tolerance of the human brain is
| less developed as human-level language is a newer
| evolutionary invention.
|
| It'd be interesting if the parameter/complexity requirements
| are actually similar once you examine the system as a whole,
| meaning machine _and_ human brain.
| Codesleuth wrote:
| > image generation models is offloaded to the human visual
| cortex which is a very old evolutionary construct and thus
| had time to become very resilient
|
| This is a very important point. A group of my colleagues
| (who are not tech people) are much more impressed with the
| image generation models than with the chat interface, even
| though the images are often whacky or just wrong. Yet the
| fact that it tried is impressive to them, with their minds
| managing to fill in the blanks.
|
| I wonder how this compares to how a toddler speaks vs.
| paints/draws, which is typically better in the former than
| the latter. I'm both cases, we fill in the blanks in our
| minds.
| v01dlight wrote:
| Toddler speaking gets impressive/surprising quite fast,
| whereas the drawing usually does not. The most surprising
| thing about most toddler drawings is listening to the kid
| describe it or tell you about making it.
| glomgril wrote:
| The consistency of descriptions is particularly
| surprising to me. Like you got a roughly circular
| collection of seemingly random scribbles, but they can
| tell you exactly which parts of it correspond to the
| person's nose, hair, arms, eyes, etc. And the
| descriptions seem to stay the same if you ask about the
| same picture on different days. Still not sure what to
| make of this phenomenon but it is fascinating.
| spacebanana7 wrote:
| I wonder whether video and metaverse generation models will
| be even smaller than an image model because of this
| mechanism.
|
| The mapping and motion parts of the human brain are also
| old evolutionary constructs that could error correct the
| output of models.
| jameshart wrote:
| Right - The 'palette' for text generation is smaller: just 26
| or so letters (plus some other characters), and if you put
| the wrong ones next to one another the result is garbage.
|
| There's something interesting in the fact that an image based
| system doesn't need as much complexity to capture a semantic
| model as a verbal system does; I think there's maybe a
| parallel there to the way that human minds find it easier to
| just 'visualize' some things as a basis for reasoning about
| them, but if we can't 'visualize' and instead have to 'think
| things through' it's a more intensive process.
|
| Like, GPT has well known trouble counting - ask it for five
| things and it will give you four or six. Humans can offload
| some thinking about counting to visual/spatial reasoning
| though.
| remix2000 wrote:
| I was having some gourmet creme brulee with my friend Zoe
| at the cafe near the entrepot in Aland's capital city and
| that made me realize it's but your naivete when you say
| English is contained in mere 26 letters, for there is a
| soupcon of uber important words that have diacritical marks
| in them.
| hgsgm wrote:
| The pallette for LLM is tokens not characters.
| dragonwriter wrote:
| And if it is characters (as it is for some models), its
| more than 26 of them for English. Between space, case,
| punctuation, and digits, its basically 7-bit ASCII
| without most of the control characters (newline is
| semantically important, the rest not), almost 100
| characters.
| kelipso wrote:
| The number of tokens is more in the range of 40k to 250k
| tokens.
| jameshart wrote:
| Right.
|
| Whereas for pixels there are over 16 million.
|
| How many more than 250k is that?
|
| Over 16 million.
| kelipso wrote:
| More like 256 of them, just an image is actually 3 layers
| with RGB. If you are going into combinations, I could
| easily say that triplets of tokens are actually what's
| important, and so 250k^3 is the real number of whatever.
|
| Actually now that I think about it, neural networks work
| with real numbers, so a pixel is just 3 numbers. Typical
| input for an image model would then be around 300x300x3
| values. While an input for a language model is around
| 2000 tokens, but while each token is inputted as an
| integer into the model, mathematically it represents a
| 250k length vector, so mathematically the input is 250k x
| 2000 values. So 90k vs 500M. Also pixels next to each
| other in an image are related, so you can reduce model
| size by taking advantage of that (CNNs).
| Der_Einzige wrote:
| Depends on the LLM. Character based LLMs exist, and even
| have advantages vs regular LLMs...
| flakeoil wrote:
| It's kind of similar to audio vs video. Although audio
| requires less data and processing than video, it's much more
| difficult to get the audio good than the video and if the
| audio is bad or even missing for some time, it's useless,
| while if the video is bad, stuck or missing, it's not that
| big of a deal most of the time.
|
| This is particularly true in a video conferencing situation.
| If the audio is bad, you miss out a lot. If the video is bad,
| it's not a big deal.
| devenvdev wrote:
| Deaf people would disagree :) if you talk in sign language
| on zoom missing video parts would ruin the conversation.
|
| I don't think it's about precision, in the case of audio vs
| video - if you remove all the even columns from a video it
| would be similar to reducing quality, the same can be done
| with audio - removing half of the frequencies uniformly
| will just lower the quality.
| kelipso wrote:
| That's a pretty specific case. You can get really good
| performance for a ton of tasks in video (video question
| answering, object identification and tracking, action
| recognition, etc) by just sampling a frame per second or
| even less frequently. Definitely can't do that with
| audio.
___________________________________________________________________
(page generated 2023-03-10 23:02 UTC)