[HN Gopher] Show HN: Explore large language models with 512MB of...
___________________________________________________________________
Show HN: Explore large language models with 512MB of RAM
Author : jncraton
Score : 116 points
Date : 2023-06-17 13:17 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| znagengast wrote:
| Isn't it crazy that the entirety of human knowledge can be
| condensed down to fit on an SD card.
| cjtrowbridge wrote:
| That's not what this is.
| PaulHoule wrote:
| A CD-ROM
| 1024core wrote:
| >>> lm.do("If I have 7 apples then eat 5, how many apples do I
| have?")
|
| 'You have 8 apples.'
|
| >>> lm.set_max_ram('4gb')
|
| 4.0
|
| >>> lm.do("If I have 7 apples then eat 5, how many apples do I
| have?")
|
| 'I have 2 apples left.'
|
| It's funny how it switched from "You" to "I" when the memory was
| increased.
| george_808 wrote:
| From what I can tell, this doesn't re-seed the random number
| generator, so we shouldn't expect deterministic results. A
| better test would be to examine the model's logits, or
| probabilities of the next token, across different ram sizes.
| behnamoh wrote:
| Nothing new here, just yet another wrapper around the current
| language models, and weak ones at that.
| rolisz wrote:
| Flan-T5 models perform really well, considering their size. In
| my experiments and in some recent papers, they are very close
| to 13b Llama for example:
| https://twitter.com/YiTayML/status/1668302949276356609
| yieldcrv wrote:
| its like the bar keeps getting lower and lower
| btdmaster wrote:
| Really cool, I didn't know CPU/traditional RAM was enough
| already.
|
| Though, surprising results from repl.it:
| lm.classify("unabridged", "positive", "negative")=="unabridged
| is"
| jncraton wrote:
| Thanks for pointing that out. Classification is half-baked at
| the moment. It should ultimately be restricting output to only
| appropriate labels, but right now it is simply sampling.
| Animats wrote:
| >>> lm.do("What is the population of Chicago") 'As of
| 2021, the population of Chicago is approximately 8.4 million.'
| >>> lm.do("What is the population of Shenzhen") 'As of
| 2021, the population of Shenzhen is approximately 1.3 million
| people.' >>> lm.do("What is the wavelength of blue
| light.") 'The wavelength of blue light is approximately
| 299,792,458 meters per second.' >>> lm.do("What is
| YCombinator") 'YCombinator is a programming language used
| to combine two or more languages into a single program.'
| >>> lm.do("What is asphalt made of?") 'Asphalt is made of
| sand, gravel, and other materials.' >>> lm.do("What is
| the square root of 2") '2.' >>> lm.do("How do I
| get to New York City from California?") 'You can get to
| New York City from California by taking a bus or train.'
| >>> lm.do("How can I unlock a lock without a key") 'You
| can use a combination of keys and a password to unlock a lock
| without a key.' >>> lm.do("How long should rice be
| cooked.") 'The recommended cooking time for rice depends
| on the type of rice, but generally it should be cooked
| for about 8-10 minutes per pound.'
|
| 100% wrong.
|
| This is an automated version of the Dunning-Kruger effect. You
| can ask it anything, and get back a confident wrong answer. So
| far, it hasn't replied to any question of mine with an indication
| that it doesn't know.
|
| It's a nice demonstration of the hallucination problem with LLMs.
| With a small data set, the results are usually bogus, but that's
| not detected.
| petercooper wrote:
| Even the largest LLMs with huge data sets. I asked GPT4 this
| morning very simply what relation Charles I was to Henry VIII
| and it said he was his great-great grandson, despite Henry
| VIII's children famously all dying childless.
| Translationaut wrote:
| Have you also tried the bigger models? The smaller models are
| good for assisted generation:
| https://huggingface.co/blog/assisted-generation
|
| Those models of LaMini-Flan-T5 are trained to follow
| instructions and not to recognize the truth content. You could
| train a transformer like Ernie or Vega (which lead superglue)
| on such challenging factual data. But don't expect mathematical
| correct results only from the model. Therefore you have
| langchain with other APIs.
| Blahah wrote:
| Yeah these aren't meant to know facts, just to parse language.
| Good for understanding simple instructions to automate tasks
| using external tools
| Animats wrote:
| The problem is not that the model doesn't have enough facts.
| It's that it has no clue what it doesn't know.
|
| If "don't know" came out reliably, small models for
| specialist areas would be useful. If small models just make
| stuff up, they're useless.
| [deleted]
| jncraton wrote:
| You can actually get these models to do this, but you have to
| ask: >>> lm.do(f"Answer from the context:
| What is YCombinator? {lm.get_wiki('Python')}") 'The
| context does not provide information about YCombinator.'
| >>> lm.do(f"Answer from the context: What is YCombinator?
| {lm.get_wiki('YCombinator')}") 'YCombinator is an
| American technology startup accelerator that has launched over
| 4,000 companies, including Airbnb, Coinbase, Cruise, DoorDash,
| Dropbox, Instacart, Quora, PagerDuty, Reddit, Stripe and
| Twitch.'
|
| Without being told to be grounded, the model will guess.
| However, it may be able to identify information not available
| in a provided context.
|
| One of my goals for this package is to provide a way for folks
| to learn about the basics of grounding and semantic search.
| Animats wrote:
| > Without being told to be grounded, the model will guess.
|
| Right. I understand why, but consider the underlying
| technology flawed unless there's some way to reject wildly
| wrong results. What's going on here looks like noise fed
| through layers which generate plausible-looking text from
| noise. Is it possible to detect that you're not far enough
| above the noise threshold to generate anything useful?
| >>> lm.do("What is Ycombinator? Do not guess.")
| 'Ycombinator is a mathematical formula that states that the
| sum of two integers multiplied by one are equal to
| zero.'
| scarface_74 wrote:
| Is there theoretically a method to have an LLM to not
| hallucinate and just say "I don't know" or just answer
| questions based on a certain domain of knowledge which it is
| well trained on?
|
| Could a business ever trust an LLM based chatbot as much as the
| old school chatbot where certain questions reliably give
| certain answers and fails when it doesn't know?
| sp332 wrote:
| LLM stands for "large language model". It's producing strings
| of text that at statistically similar to sequences in the
| training data. External reality isn't relevant. It's at least
| possible to run the result through a confidence filter, but
| that would be a feature of the tooling, not the model.
| Llamamoe wrote:
| Well, this kills my enthusiasm to actually play with it. That's
| bad enough I'm not sure it can even be used as anything but a
| glorified Markov chain.
| xrd wrote:
| At first glance i would be surprised if this works at all.
|
| The readme says it loads a significant amount of data the first
| time, 250mb. None of the LLM weights I know of are less than
| several gigabytes in size.
|
| It says it only requires 512mb of RAM. None of the interesting
| LLMs I know of run in less than 6gb of VRAM.
|
| It says it uses no API keys, which is great, but that means
| inference is local, which I can't imagine works with the above
| constraints.
| nattaylor wrote:
| At 512gb RAM, it uses https://huggingface.co/jncraton/LaMini-
| Flan-T5-248M-ct2-int8 which says
|
| "This model is one of our LaMini-LM model series in paper
| "LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale
| Instructions". This model is a fine-tuned version of
| google/flan-t5-base on LaMini-instruction dataset that contains
| 2.58M samples for instruction fine-tuning"
| Translationaut wrote:
| Those minified models are still equal or bigger compared to the
| initial "attention is all you need" transformer.
| jb1991 wrote:
| Shoot! I only have 511MB of RAM. Time to upgrade and get that AI!
| GaggiX wrote:
| In case anyone is wondering, this repo uses various fine-tuned
| Flan-T5 models.
| rolisz wrote:
| And Flan-T5 models are awesome! Fastchat is a finetuned 3b
| parameter Flan-T5 model that can perform really well,
| comparably to llama 13b
| jncraton wrote:
| That's correct. The current base model is an int8 quantization
| of LaMini-Flan-T5-248M described here:
|
| https://github.com/mbzuai-nlp/lamini-lm
|
| I shared more details over on Reddit:
|
| https://www.reddit.com/r/LocalLLaMA/comments/14btk3a/explore...
___________________________________________________________________
(page generated 2023-06-17 23:02 UTC)