[HN Gopher] The Coming of Local LLMs
___________________________________________________________________
The Coming of Local LLMs
Author : yarapavan
Score : 192 points
Date : 2023-04-11 15:21 UTC (7 hours ago)
(HTM) web link (nickarner.com)
(TXT) w3m dump (nickarner.com)
| Sparkyte wrote:
| Everyone be like ChatGPT is everything. I am like it won't be the
| last or the first.
| sharemywin wrote:
| I could certainly see them being trained/used as a front end for
| a database(file system, api calls).
| ingenieroariel wrote:
| In order to run _large_ language models, we should all be buying
| a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of
| GPU and Neural cores.) and putting Linux on it to remove the
| artificial restrictions.
|
| Yes, we will be running them soon in low end hardware, but we
| need to get at least to GPT-3.5-turbo level of inference speed
| and quality before we try to make it small.
|
| I already started. $ neofetch
| -` x@decpti .o+`
| ------- `ooo/ OS: Arch
| Linux ARM aarch64 `+oooo:
| Host: Apple Mac Studio (M1 Ultra, 2022)
| `+oooooo: Kernel: 6.1.0-rc6-asahi-4-1-ARCH
| -+oooooo+: Uptime: 4 hours, 23 mins
| `/:-:++oooo+: Packages: 177 (pacman)
| `/++++/+++++++: Shell: bash 5.1.16
| `/++++++++++++++: Resolution: 1920x1080
| `/+++ooooooooooooo/` Terminal: /dev/pts/0
| ./ooosssso++osssssso+` CPU: (20) @ 2.064GHz
| .oossssso-````/ossssss+` Memory: 717MiB / 129540MiB
| -osssssso. :ssssssso. :osssssss/ osssso+++.
| /ossssssss/ +ssssooo/-
| simlevesque wrote:
| Please, this is not r/unixporn.
| beiller wrote:
| Which LLM can run on apple's neural cores / GPU cores? I can
| only run on plain ol' CPU cores (llama), and it runs fine on my
| Ryzen CPU for less than half the price of that system. That
| being said I'm switching from Ubuntu to Arch cause I'm sick of
| all my packages being way out of date!
| ingenieroariel wrote:
| tinygrad
|
| https://github.com/geohot/tinygrad/tree/master/accel/ane
|
| But I have not tested it on Linux since Asahi has not yet
| added support.
|
| Same machine but OSX, llama.cpp runs at 18ms per token (7B)
| and 200ms per token (65B) on CPU using float16.
| monkeydust wrote:
| Wait till these are embedded into soft toys.
|
| 'You are a koala who plays with 5-7 year olds, you are friendly
| natured and curious and like to ask questions'
| jjtheblunt wrote:
| Like Teddy from A.I. the movie?
|
| https://www.youtube.com/watch?v=YRsICbxDEiI
| waffletower wrote:
| While many NLP related Apple ML job listings have been added
| since this article was written, there were several recent
| listings at the time of its writing. While I feel that Apple does
| not focus well on intangible technologies, products that can't be
| readily carried, worn and given their boutique product
| development fetish focus, I have some hope that they can overcome
| this bias somewhat, and see how behind they are.
| hackernewds wrote:
| I disagree with the assessment that Apple is behind. Apple is
| known for executing well and putting their weight behind the
| things they launch.
|
| Like Jack Dorsey would often say "it's not important to be
| first to market, you can just be best to market". And the world
| got CashApp.
|
| I'm sure however Apple enters the space, it will be fleshed out
| (vs Bard).
| rootusrootus wrote:
| I want to agree, but it's pretty easy to find instances where
| Apple has dabbled but not delivered best-in-class solutions.
| Siri. iCloud. Home automation.
| pwinnski wrote:
| Apple has two values in conflict with each other, I think.
| On the one hand, they want to deliver best-in-class
| solutions. On the other hand, they have a commitment to
| user privacy[0] as perhaps only a gay man growing up in the
| south might value.
|
| Siri should be better! It _lost_ features post-acquisition
| by Apple, and it seems like user privacy is why.
|
| Home automation is arguable. If you consider a single point
| of failure on a server somewhere to be bad, Apple's
| solution is pretty great. Their commitment to zigging where
| others zagged put them behind, since hardware vendors
| didn't want to put in powerful (expensive) enough chips to
| handle the cryptography, but while other companies go out
| of business, or transmit images and video to external
| parties, Apple's works reliably and securely.
|
| Still, as with most of my complaints about Apple, it's a
| trade-off between privacy and functionality, and Apple will
| seemingly always choose privacy over functionality, even as
| Google consistently chooses functionality over privacy.
|
| [0] Yes, there are examples of edge cases that suggest a
| less-than-perfect record. Contrast that with their
| competitors, for which invading privacy is foundational to
| the business model.
| sliken wrote:
| Up until recently I'd have agreed on Apple's.
|
| Increasingly Apple seem to be blocking tracking, noteably
| from facebook, to make the most profit of that tracking.
| I've read claims that apple made between $5B and $20B on
| advertising in 2022. It's far from clear that Apple's
| view on privacy is going to stay the same.
| whimsicalism wrote:
| Apple has very little ML expertise and very few cloud
| resources.
|
| They are absolutely behind in the space and anyone who works
| in the industry will tell you that. Only feasible way IMO
| would have to be a very big budget acquisition of one of the
| major LLM startups, but most of those already have big tech
| backers.
| layer8 wrote:
| It would be nice to be able to run an LLM-driven spamassassin on
| a VPS for acceptable cost.
| londons_explore wrote:
| I don't think it will help. Actual friends occasionally send me
| mail that says "test" from a random account. And spammers do
| too... There is no way to seperate them.
| layer8 wrote:
| Personally I haven't had that problem. I can readily
| distinguish spam from non-spam by manual review ~99% of the
| time. If I could train or instruct an LLM to do the same as I
| do now, I would be happy. My current false-negative rate with
| Bayesian spamassassin is more like 50%.
| 0xDEF wrote:
| I know it's secret how GPT-4 actually works but are there any of
| these local LLMs that are also multimodal?
| whimsicalism wrote:
| No
| txomon wrote:
| Out of doubt, which seems to be spreading around the internet.
| The LLaMa model weights weren't "leaked" AFAIK but rather
| explicitly given access to to researchers, isn't it right?
|
| I know the article goes on to speak about something else, but I'm
| not sure why this claim that the LLaMa model weights were leaked,
| as in unintendenly made available is being done.
| turmeric_root wrote:
| The model weights were only shared by FB to people who applied
| for research access. Github repos containing links to the model
| weights have been taken down by FB.
| ftxbro wrote:
| My understanding is that researchers could ask for access to
| weights, but then also they were leaked so that anyone could
| get them without asking. There is another layer, where Facebook
| seems to accept it on some level (I mean they don't have a
| choice anymore anyway); they put a cheeky comment in the open
| pull request instead of closing it.
| binkHN wrote:
| This is wonderful. As hardware and software continues to improve,
| everything seems to find a way to run on ever smaller devices.
| Guess your own pocket-AGI is not too far away after all.
| 1827162 wrote:
| By the way, I was thinking of something along the lines of a
| powerful FPGA with direct access to large quantities of very
| fast NAND flash, likely many chips in parallel, which will save
| having to load the model into RAM..... So it will be able to
| directly run from NAND flash, which opens up the possibility of
| using very large models???
|
| Power consumption would not be an issue if it's used
| sporadically throughout the day, it's not like it needs to run
| continuously?
|
| There is still the issue of NAND flash read disturb, which I
| haven't fully looked into yet.
| [deleted]
| jrm4 wrote:
| As I see these things come out, it feels like there's not a lot
| of discussion on _which hardware_ (that isn 't one of the fancy
| new Macs?) As in, there might be a lot of graphics cards out
| there that could be used here? Is it only Nvidia still, is AMD a
| possibility? Maybe I'm missing something on how the tech works?
| [deleted]
| seydor wrote:
| This has a list of models and their VRAM requirements
|
| https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...
| anentropic wrote:
| Apple should get working on a version of the Neural Engine that
| is useful for these models, and remove the 3GB size limit [1] to
| take full advantage of the 'unified' memory architecture. Game
| changer.
|
| Waste of die space currently (on Macbook at least, I'm sure they
| find uses for it in the iPhone)
|
| [1] https://github.com/smpanaro/more-ane-
| transformers/blob/main/...
| emrah wrote:
| It's great they got LLMs running on resource constrained devices
| but are they any good? Or I should ask, with the limited
| resources they get, what good are they for?
| BulgarianIdiot wrote:
| They're crude but will be getting better quickly.
| cgearhart wrote:
| Even with smaller models & more optimized hardware, I think
| edge compute is going to be power-limited first. Batteries
| today just won't support constantly running LLMs. But I joked
| recently that as long as they prove useful then consumers
| would be willing to swap their iPhone for the old car battery
| with a phone handle attached.
| BulgarianIdiot wrote:
| The first step is for it to be viable for smaller models on
| desktops. The rest will follow, as hardware catches up.
| Hybrid analog-hybrid NN hardware is on the horizon, maybe
| in 3-4 years. This would allow GPT-4 level performance on
| an iPhone with plausible battery life.
|
| The current hardware of course can't pull anything like
| this yet. But iPhone supports on-device facial recognition,
| object recognition, dictation and translation, so small
| steps...
| tudorw wrote:
| I guess if the hardware is cheap for some uses speed is not so
| important, you can just walk away and let it grind.
| kbrkbr wrote:
| From my experience with llama.cpp and oobaboogas webui I can
| say they are amazing, at least on my gaming pc. I'm absolutely
| astonished at the speed and quality of llama, alpaca, galactica
| and vicuna (the >10B parameters ones).
|
| Make no mistake, it's for tinkerers that do not expect each
| prompt to be answered human like.
|
| I see them as creativity and thought testing tools, also
| knowledge exploratory.
| [deleted]
| synergy20 wrote:
| are we talking about training or inference for local LLM
| here? it's hard to do any meaningful training on the edge
| unless we all carry a heavy gaming pc, even that, the
| training quality will be subpar?
| qumpis wrote:
| Inference, even fine-tuning a few layers would be difficult
| since one needs to use non-quantized model, I'd imagine
| ctoth wrote:
| Checkout LoRA and Alpaca LoRA and the whole huge group of
| people who have already figured this out. I think there
| was another breakthrough (yesterday?) which is a further
| adaption of LoRA to touch even less parameters at
| runtime.
| homarp wrote:
| DyLoRA https://news.ycombinator.com/item?id=35514228 is
| what you meant
| whimsicalism wrote:
| Disagree that this is a major breakthrough :) but it is
| probably a marginal improvement.
| whimsicalism wrote:
| They're not training on edge :)
| whimsicalism wrote:
| If they are amazing, then ChatGPT must be God-like in your
| view.
|
| I've been underwhelmed by Alpaca and Alpaca-LORA and LLaMA
| all at 13B but I have not tried higher params.
| yyyk wrote:
| 30B Alpaca is better. Not anywhere near ChatGPT level, but
| better than the 13B models.
| kbrkbr wrote:
| In my opinion the problem with these is engineering a good
| prompt. I read of lots of people only getting nonsense or
| repetitions, and learned a bit from what they shared. These
| models are not chat bots.
|
| Vicuna is more friendly in that regard.
|
| But I'm well aware of their limitations also, and I can see
| how one can be underwhelmed. They are not jacks of all
| trades
| whimsicalism wrote:
| Alpaca & Alpaca-LoRA are literally trained to be chat
| bots. Their goal is to be instruct-tuned.
| wing-_-nuts wrote:
| is there a difference in the quality of llm one would be able
| to train or run on a gpu with 8, 12, 16, all the way up to
| 24gb?
|
| I'm trying to decide whether it's worth while to splurge on a
| more expensive 4090 vs a 4070 or whatever.
| Garrrrrr wrote:
| llama.cpp runs on your processor and uses a lot of RAM, so
| splurging on a GPU isn't going to help the performance
| whimsicalism wrote:
| It makes 0 economical sense to buy a GPU to train a model.
| If you want to train a model, train it on the cloud.
| anonzzzies wrote:
| Can you calculate that for me on a napkin? Every
| calculation I make for training, but certainly for
| inference, makes me break even after well under a year
| and then it's vastly less if I buy the hardware myself.
| whimsicalism wrote:
| You can infer on your local machine.
|
| I doubt it will take you a year to train a model.
|
| Vicuna-13B cost $300 to train/fine-tune [0]. They trained
| on an A100 which costs $10k [1].
|
| [0]: https://vicuna.lmsys.org/ [1]:
| https://github.com/lm-
| sys/FastChat/blob/main/scripts/train-v...
| wing-_-nuts wrote:
| It's more that I'm building a gaming pc this summer, and
| I can either target 1440p (4070) for 2k or 4k for 5k
| (4090). If I can do a lot more with a 4090 over a 4070 it
| might make sense, but I know a lot of cs students use
| google colab these days, so I may just rely on that.
| thewataccount wrote:
| I'd seriously recommend the 4090 over the 4070 if you
| want to do finetuning/inference locally. And I highly
| recommend 64GB of ram.
|
| The 24GB of VRAM is 100% worth it alone. If you want to
| do local ML stuff you _need_ that 24gb of VRAM.
|
| 64GB of ram + 24GB of vram lets you run a lot of the
| medium size models at decent speeds. I don't use Colab
| personally but AFAIK it should work fine for you if you
| don't want to do it locally.
|
| Also worth noting is the newer ray tracing rendering that
| cyberpunk is doing. You should checkout the demos IMO it
| looks sick. It only runs at 18fps on a 4090 so it's only
| playable on a 4090 + dlss, and I'm not sure if the newer
| rending tech will be super achievable on any of the other
| cards - if that's of interest to you.
| whimsicalism wrote:
| Fair enough.
| thewataccount wrote:
| This isn't necessarily true with LoRAs - a 4090 can
| train/compute the alpaca dataset with LoRA in under 6
| hours (it might be 3, I forget what it was).
|
| So finetuning with LoRAs and a few other methods is fine
| on higher end consumer hardware like a 4090 and finishes
| in a reasonable amount of time - IMO definitely worth it
| if you're experimenting with this especially for the
| inference.
|
| The base training though yeah I totally agree with you -
| train in the cloud, don't buy hardware when you need a
| month of 8x A100's or whatnot.
| whimsicalism wrote:
| Even with LoRA the economics are not in your favor to buy
| a 4090 instead of cloud training.
| thewataccount wrote:
| Actually yeah you're definitely right.
|
| My perspective was for people who have other uses for
| them e.g. gaming or local inference. From a pure finance
| standpoint you're definitely right - you should rent and
| not buy a dedicated card. I think you'd need a few
| thousand hours to break even which is a few months 24/7.
| turmeric_root wrote:
| More VRAM => larger models. IME it is absolutely worth
| maxing out VRAM for the significant improvement in quality,
| especially with LLaMA (though even with a 4090, you won't
| be able to run the largest 65-billion parameter model even
| with 4-bit quantization).
|
| That said, I recommend renting a cloud GPU for a few hours
| and trying the larger models on them before buying a GPU of
| your own, just to see if the models meet your requirements.
| sliken wrote:
| But should fit easily on a Apple MBP or Studio with 96GB
| or 128GB of unified memory.
| rubidium wrote:
| Is there a "getting started" guide you'd recommend to a
| newbie to the space?
| kbrkbr wrote:
| I started with ggerganov's llama.cpp GitHub repo, and went
| from there. But then again, I know some programming,
| statistics and machine learning, so it may not be for you,
| I cannot judge that.
|
| Models can be found on huggingface.co, and I'd start with
| eachadea/ggml-vicuna-13b-4bit, but it needs 10G of cpu-ram.
| It is very friendly to any prompt though.
|
| I read on my way (on reddit, when I recall correctly), that
| there must be some really good intro videos on YouTube.
| turmeric_root wrote:
| I like using them for memeing
| gigel82 wrote:
| I don't understand why people are so excited to build this big
| thing on top of Llama, which is closed source, severely license
| restricted and we now know for a fact that Meta is going after
| users with the legal hammer.
|
| I'm sure if we'd pool resources together we could build a truly
| open alternative worthy of building on top of.
| whimsicalism wrote:
| > I'm sure if we'd pool resources together we could build a
| truly open alternative worthy of building on top of.
|
| This would require like ConstitutionDAO level of resource
| pooling without direct monetary payoff.
|
| I mean, good luck.
| whimsicalism wrote:
| It appears there is this genre of articles pretending that LLAMA
| or its RL-HF tuned variants are somehow even close to an
| alternative to ChatGPT. Spending more than a few moments
| interacting even with the larger instruct-tuned variants of these
| models quickly dispels that idea. Why do these takes around open-
| source AI remain so popular? What is the driving force?
|
| I've posted this before, but it seems like this genre is just
| getting more and more popular - and more and more untethered from
| any actual metrics of how good these models are.
| seydor wrote:
| You make it sound as if ChatGPT itself isnt severly limited.
| Vicuna and newer 13B models are quite close . And the
| uncensored models have a capability that ChatGPT will never
| have.
| causi wrote:
| The way OpenAI has set their models up for prompting and
| follow-up is where their real advantage is. It's very hard to
| take a model you can run locally and just go "Hey, write me a
| powershell script to convert all these files into this
| format. Split them up so they're no more than an hour each.
| Ok, change it to also denoise them."
| thewataccount wrote:
| Are you talking about the RLHF? That's where Vicuna/Alpaca
| and other finetuning help a lot.
|
| There's also already tools for conversation flows (which
| just means you prepend the conversation history to the
| prompt).
|
| I'm not saying the performance is nearly as good, but the
| actual workflow does already exist and is massively
| improving. The interesting part to me is that this
| finetuning can be done in a couple few hours on a consumer
| gpu (4090).
| whimsicalism wrote:
| They just aren't quite close. Maybe you guys are asking
| shallow questions, but I'm asking subject matter questions
| and this is just not true.
|
| It would be great if we could have detailed QA evaluation to
| show this, but of course then the open source people would
| train their models on it as a fine-tuning datasaet.
| UncleEntity wrote:
| Train it on specific subject matter and not waste space on
| things like "what year did Jack Nicholson beat on that
| dude's car with a golf club?"
|
| I mean... the horror.
| whimsicalism wrote:
| That's just not how these models work.
| taneq wrote:
| You'll have to forgive them, they're probably trained
| mostly on HN posts. ;)
| avereveard wrote:
| What are the newer models? I am testing them in batteries
| across complex tasks, so far vicuna is the most flexible but
| they all choke on reflective instructions (I. E. Knowledge to
| extract is not in the model but in the user text)
| ExxKA wrote:
| I very interested in this usecase - where should I go on
| the interwebs?
| avereveard wrote:
| original paper: https://arxiv.org/abs/2205.00445
|
| langchain agents are a good starting implementation.
|
| you can build your own prompt and get the ai to work by
| iself hallucinating tools, which may be cheaper to test
| out than going back and forth with an agent manager. not
| as accurate, but you can still extract useful work, i.e.
| https://i.imgur.com/AE4R3dR.png (gpt-35-turbo is
| traditionally failing this task completely, prompt get it
| to work at it)
|
| these prompt all require the model to work off data
| within the prompt within the first shoot. model require a
| degree to introspection for that to work.
| eikenberry wrote:
| I'm interested as LLMs are interesting tech but ChatGPT is
| proprietary and useless for free software.
| gitfan86 wrote:
| Because OpenAI or Google being the gate keepers is a show
| stopper for most serious people in the space.
|
| They will be able to shutdown your startup on a whim. Even if
| they didn't politicians and regulators would be a huge risk.
| Without democratization we get blade runner . Not that
| democratization has no problems, just that it is the way a lot
| of us are wanting it to go.
| whimsicalism wrote:
| And basing your startup on a model pirated from Meta isn't
| risky?
|
| I get it, I understand why people like decentralization - but
| the open source community doesn't even have close to the
| capability to train an actually open-source LLaMA equivalent.
| xyzzy123 wrote:
| Mozilla or Wikimedia foundation have budgets with the right
| number of zeroes to support an effort like this.
|
| While those organizations might not be the right fit, I
| think they serve as an "existence proof" that a large scale
| open / nonprofit project is not completely inconceivable.
|
| Could possibly see an industry consortium (of players too
| small to compete on their own) funding an open effort.
|
| Last idea sounds crazy but hear me out: how much would
| Nvidia spending $100M on "open" models boost spending on
| graphics cards? I hope someone's running the numbers on
| that...
| whimsicalism wrote:
| > how much would Nvidia spending $100M on "open" models
| boost spending on graphics cards?
|
| Probably not as much as in a zero-sum game where everyone
| is trying to train their own model. Every leap in CPU
| inference makes this an increasingly less appealing
| option for them.
|
| But I agree, some industry consortium might try to do it.
| I think they would first have to be relying heavily on
| LLMs before they were willing to do that, and its
| possible by then that the lead will have gotten too large
| to easily surmount, especially now that everyone has
| stopped publishing.
| ingenieroariel wrote:
| They are comparable to the first ChatGPT proof of concept from
| say early 2022. The reason many of us are excited is because we
| may be a year or two away from being able to run a ChatGPT if
| the open source models follow a similar curve.
| whimsicalism wrote:
| I have been using ChatGPT from the beginning :) ChatGPT did
| not have a PoC in early 2022.
|
| They are not comparable with late 2022 ChatGPT.
| simonw wrote:
| I assume you mean GPT-3 - ChatGPT didn't launch until
| November 30th 2022.
| simion314 wrote:
| People are enthusiastic about the possibilities, imagine a
| black box on your desk contains only RAM and matrix
| multiplication chips, you install on it your favorite AI
| assistant and you train it with your private data/code, you
| remove all prudish restrictions and get productive on your work
| and on your off times.
|
| Llama has the potential to reach ChatGPT it needs tunning to
| get better at responding to questions, llama if I am not worng
| is mostly attempting to predict what is next.
|
| I can see it similar like Midjourney and Stable diffusion,
| midjorny can make any stupid prompt look like a digiatal art in
| the style of Midjorney but look how many stable Diffusion
| innovation happens, a competent person that is on top with all
| the new stuff can produce absolute anything in any style they
| want.
| alchemist1e9 wrote:
| My hunch is model weights will be commercialized as a
| purchased object. They can be watermarked so easy to trace
| any leak.
|
| Then hardware will be a separate business. I think Apple
| might be caught off-guard by Nvidia on hardware. The latest
| NVlink and 400Gbps interconnects when combined with H100 next
| iterations and also rumors of advanced PCIe motherboards with
| high lane Nvidia CPUs and it looks to me that next year they
| can be selling $100-300K physical systems optimized for LLM
| inference that physically remind me of mainframes.
| simion314 wrote:
| It makes no sense, the science is free and open, the
| companies just put their money into throwing data into the
| model. Once you have the big llama model filled with all
| the humanity information as an open source thing at bast a
| company could sell you some small stuff to add on top, like
| maybe Disney would sell you a "license" and a lora to
| generate Disney crap, their model would probably will be
| lower in quality then the open ones but the license would
| be the important part.
|
| It is kind of idiotic that some scientist can spend years
| and a lot of public money to create some technology and
| then bilionairs are miliking all the profits.
| alchemist1e9 wrote:
| I definitely hope you are correct.
|
| I'm really hoping there are viable distributed and
| somewhat decentralized eventually consistent training
| algorithms we could all run in a P2P system. That would
| be super cool.
|
| However I can easily see that now the framework has been
| established if a company builds a proprietary curated
| dataset for specific skills and then pays to spend
| resources on specialized reinforcement training.
|
| Then they can commercialize that I would think. As people
| would pay for an LLM that does XYZ the best. Kinda like
| your Disney example but I was thinking engineering tasks
| in my head.
| whimsicalism wrote:
| I am similarly enthusiastic and super, super impressed by
| Llama.cpp
|
| But I just don't see the need to mislead and suggest that
| these models are "on par" with ChatGPT or something like
| that. They just aren't.
| simion314 wrote:
| I did not said that are the same, the llama has a lot of
| information in it , the issue seems to be the Q&A chat
| part. This can be added on top by the community without
| having to start from scratch IMO. But I might be wrong, and
| OpenAI put some magic shit in , some super secret
| unpublished stuff, in that case educate me. In my mind I
| compare with Stable Diffusion, the proprietary ones produce
| more artistic effects because they put some more stuff into
| the prompts and they are imposing their styles. With SD you
| have the control and you need to wait for some
| improvements, plugins, new loras or embedings with new
| styles etc. In SD I could train my face in 1- minutes, with
| proprietary shit it will never happen...
|
| Imagine what a math community could train, they just need
| access to the model and soem GUI software that can help
| them train.
|
| So llama based Chat stuff is not yet comparable with
| ChatGPT but there are already lot of progress made. At this
| moment coding and math is bad in llama based but other
| stuff is great, like story creation, also I only could test
| 3-b 4bit and it is good enough to for example provide me a
| complex response in valid JSON format.
| transfire wrote:
| We really need optical computing to take this to the next level.
| cs702 wrote:
| I expect we will see the biggest jump in performance if (when)
| consumer-grade coprocessors like mobile GPUs start incorporating
| _attention layers_ as a primitive building block at the hardware
| level, e.g., with instructions and memory layouts engineered
| specifically to make ultra-low-precision (say, 4-bit) transformer
| layers as compute- and memory-efficient as possible on consumer
| devices. _That_ seems almost inevitable to me.
| liuliu wrote:
| Chip's capability planning seems need about ~2yr lead time. So
| we are expecting fastest would be somewhere around end of 2024.
| (Transformers probably earlier than that (2023?), 4-bit would
| be later).
| cs702 wrote:
| Unless there are groups that have been thinking about and
| working on this for a while...
| zamalek wrote:
| There are those Google accelerators that plug into an M.2 slot.
| You could plausibly do this today, although I am not sure what
| sort of memory constraints those accelerators have.
| beauzero wrote:
| https://www.electronicsweekly.com/blogs/gadget-
| master/boards...
| BrutalCoding wrote:
| These ones can be plugged in with USB type-c, see:
| https://coral.ai/products/accelerator/
|
| It's used for boosting interference (offline) on Linux, Mac
| and Windows.
|
| Haven't bought or used them but I've had my eyes on these for
| a little while!
| jjoonathan wrote:
| Low precision: agreed
|
| Attention: Isn't it quadratic in context length? I dunno, this
| feels like the crude first iteration of something that will get
| inevitably passed by something that scales better.
| throwawaymaths wrote:
| In theory one could use the nysromformer. Don't know if
| anyone does in practice
| whimsicalism wrote:
| They don't
| throwawaymaths wrote:
| Do you know why/why not?
| whimsicalism wrote:
| They've just gone with a different approach for fast
| attention. [0] Not sure it was due to any particular
| merits/lack of the nystroformer.
|
| [0]: https://twitter.com/typedfemale/status/1609867110695
| 735296
| dpflan wrote:
| Sharing a comment from a similar line of discussion:
|
| """
|
| Complexity is quadratic in sequence length. For 512 tokens it
| is 262K, but for 4000 tokens it becomes 16M and goes OOM on a
| single GPU. We need about 100K-1M tokens to load whole books
| at once.
|
| Since 2017 there have been hundreds of attempts to bring
| O(N^2) to O(N), but none of them replaced the vanilla
| attention yet in large models. They lose on accuracy. Maybe
| Flash attention has a shot
| (https://arxiv.org/abs/2205.14135).
|
| """
|
| - source: https://news.ycombinator.com/item?id=34171503
| whimsicalism wrote:
| There is literally no way that GPT-4 is using classic
| O(N^2) attention exclusively. They haven't released the
| results, but I promise you this is not what they are using
| exclusively.
| sebzim4500 wrote:
| Probably once context lengths get really long it might be
| better to use ANN rather than exact attention at inference
| time. I would imagine this would only pay off with ~100,000
| token contexts, and even then it would only work if only a
| few tokens meaningfully contribute to each attention head.
| cs702 wrote:
| Memory use is already linear in practice, thanks to
| FlashAttention. It's an open question whether computation can
| be made sub-quadratic without impacting model performance,
| although there are ongoing efforts seeking to do exactly
| that.[a]
|
| Keep in mind: Once you go into precision as low as 4 bits (or
| lower?), all sorts of optimizations can become practical. Off
| the top of my head, maybe you could cache and reuse common
| attention sub-matrices (e.g., a 16x16 sub-matrix with 4-bit
| elements occupies only 16x16x4/8=128 bytes of space)?
|
| My sense is there's so much money at stake here, that whoever
| does this first will win big even if they end up having to
| replace or augment it with something better down the road.
| Hypothetical example: Imagine Intel or AMD coming out with a
| $1K or $2K card that has "built-in 4-bit attention," enabling
| you to run transformers of much greater scale on a run-of-
| the-mill desktop PC. I'd buy that in a heartbeat.
|
| [a] Here's a recent post about a new approach from a group at
| Stanford that looks promising to me, although I don't fully
| understand all its details yet:
| https://news.ycombinator.com/item?id=35502187
| whimsicalism wrote:
| I agree - unfortunately we have gone so far down the rabbit
| hole optimizing transformer models.
|
| Alternative models like S4 have been able to get transformer
| level performance with O(N) sequence length scaling.
| [deleted]
| seydor wrote:
| maybe one could use physics to do that in analog. Or even
| better, in biology. I think with a clump of ~1.5kg of neurons
| we can have a pretty efficient coprocessor that is fed with
| pizza.
| bick_nyers wrote:
| On the other hand, securing the biological coprocessor in
| your homelab and running queries on it 24/7 appears to have
| legal implications.
| cs702 wrote:
| Moral implications too. The first Matrix movie is about the
| moral implications.
| mcculley wrote:
| In polite company it is considered rude to suggest that
| there was more than one Matrix movie.
| la64710 wrote:
| One simple thing these LLM models cannot do yet .. that is to
| simply point a LLM to a URL and it will start scraping - ie
| follow the hyperlinks and start consuming the content. I am not
| an AI guy but I guess this has to do with the context limitations
| of most model? How did they train OpenAI with all internet data
| till 2021? This I think will be a most popular feature for LLM
| models and I seriously hope it is OSS whenever it comes out.
| marijnz wrote:
| This already works with ChatGPT plugins:
| https://openai.com/blog/chatgpt-plugins
| yacine_ wrote:
| I was able to run a LLaMa on my personal machine to run some
| labeling on my documents, as a test of its capabilities. It was
| instruct tune. 30b parameters
|
| 4 example labels, and I had a binary classifier in seconds. Sure,
| semantic text classifiers were possible for a while, but making
| it accessible changes everything. Giving anyone who can use a
| spreadsheet the power of a local LLM (or, basically free LLMs)
| can make them much, much more productive. A lot of office work is
| clicking through sheets and doing manual labeling.
|
| It's truly wild what is becoming accessible! Really excited to
| see the next gen software that the open community comes up with
| :)
| rcme wrote:
| LLMs as general purpose classifiers is a really big deal,
| especially because you can give them fuzzy instructions. I know
| people are worried about LLMs and spam, but I think LLMs may
| provide an opportunity to elevate online discourse by being
| more efficient at filtering out spam and low quality
| commentary.
| nathan_compton wrote:
| Like almost all machine learning stuff, I expect these things
| to only be useful in places where it doesn't really matter if
| the results are correct. When you apply a classifier in real
| life its critically important to understand its statistical
| characteristics which is typically done via model
| characterization, which involves cross validation or boot
| strapping or whatever. I think the idea that you can just
| zero-shot or few-shot deploy these things as classifiers and
| forget about it is incredibly naive unless (as I said
| earlier) the results just don't really matter.
|
| I've already used LLMs in my work as a data scientist but it
| requires a ton of work to just make the results tractable
| (and I have been using GPT4, which behaves pretty well).
| These smaller language models ain't so regular. Ok, like
| consider a basic thing you want to do with a classifier:
| understand its behavior on a held out data set. Since no one
| knows what is really in the training data (since its so
| large), its quite hard to understand what the model can
| generalize about and what it has just accidentally memorized.
| CF reports that GPT4 doesn't perform nearly as well on even
| simple programming exercises that are chosen in such a way as
| to be sure they weren't in the training data.
|
| There is enormous potential for statistical fuck ups here.
| Prompt engineering, for instance, is an easy place for over-
| fitting to happen as a prompt is fine tuned on data the
| prompt engineer has and thus fails to generalize to new data.
|
| I do think there is a lot of value here, but I'm also sure
| that sloppy use of large language models is going to cause a
| bunch of trouble in the short to medium term, generate a lot
| of garbage, pollute a lot of databases, etc, while we figure
| all this stuff out.
| rcme wrote:
| Nothing about a zero or few-shot classifier precludes you
| from validating on a data set.
| nathan_compton wrote:
| You can certainly hold out a validation set while you are
| writing your prompt, but you can't know whether the model
| is over fitting for your data set since you don't know
| what data was in the training set.
| rcme wrote:
| You'll probably find out pretty quick in production :)
|
| But I posit that most text classification tasks don't
| have such strict accuracy requirements. For one, no text
| classifier is 100% accurate. For instance, I have genuine
| mail in my spam folder frequently. I see spam on social
| networks, etc. I struggle to think of cases that aren't
| at least somewhat tolerant to some amount of incorrect
| classification.
| muyuu wrote:
| spam and especially phishing will become much, much better
|
| it will be hard to trust anything at all
| SV_BubbleTime wrote:
| Which will force drive signed communications. So... win.
| muyuu wrote:
| Yea in a sense it will be a win, but it will up the
| stakes. A lot of people with get scammed. A lot more than
| right now.
| yacine_ wrote:
| very simple, just don't read the email
| akiselev wrote:
| That's great. My coworkers are already a step ahead on
| security!
| muyuu wrote:
| I'm half way there, I already have no mates and I ignore
| most of the email from work.
| gamegoblin wrote:
| I already have a custom browser plugin that calls out to GPT
| (gpt-3.5-turbo is cheap and good enough for this) to classify
| and filter out low-effort, overly negative, or intellectually
| dishonest HN comments. It significantly improves the
| experience on this site.
|
| Bonus points: I had never written a browser plugin, but GPT4
| helped me do it in under half an hour.
| darkgreene wrote:
| Do you have a repo you can share? I'd be very interested in
| running my own local copy
| tough wrote:
| +1
| ChatGTP wrote:
| Some negative, even overly negative comments contain good
| ideas though so maybe it's not a great idea?
| jpe90 wrote:
| This sounds super cool, I'm very curious how you
| implemented it.
|
| I'm only vaguely familiar with the API.. if I had to guess
| I would say you send
|
| - a system instruction that its job is to filter unwanted
| content
|
| - examples of unwanted content
|
| - an instruction like "filter the following html:"
|
| For every web request you want to filter, you would re-send
| all of those messages followed by the page HTML as the
| final message. Is that close?
| gamegoblin wrote:
| Close, but it's a bit more specialized to just work on
| HN:
|
| - Examples of unwanted content
|
| - Then I give a large numbered list of comments and ask
| which numbers should be filtered
|
| - The plugin then just deletes those comment nodes from
| the DOM. If HN ever updates their HTML I will have to
| tweak this code.
|
| The reason to send a large list of comments is just to
| save on costs. It's cheaper to do it this way than one
| comment at a time.
|
| So the main difference from what you've proposed is GPT
| never sees the HTML. My code enumerates the comments in
| the HTML and splices them in to the prompt in a nice
| numbered list, then does the reverse translation from
| list number to DOM element in the other direction.
| jpe90 wrote:
| Oh that's clever! Thanks for answering!
| artificial wrote:
| This sounds great, would love to see some more about this
| endeavor.
| metalliqaz wrote:
| How do you define the cost function for 'low effort'?
| gamegoblin wrote:
| I give GPT some examples of what I consider low-effort
| comments and non-low-effort comments, and then I just ask
| it to do classification. It's quite good at few-shot
| classification of fuzzy stuff like this.
| xk_id wrote:
| I don't know how anyone can trust to run code on their
| machine that they don't understand and hasn't been reviewed
| by a third party.
| sliken wrote:
| How many line of codes are in the kernel, drivers, and
| apps on your laptop/desktop?
|
| How sure are you that all that code has been reviewed by
| a 3rd party? How many CVEs a year impact your
| laptop/desktop?
|
| Do you have any reason to think that increased
| productivity with LLM assistance will result in lower
| quality code? Personally I find LLM assistance increases
| productivity, decreases the penalty of using a more
| difficult language like rust, and makes it more palatable
| to spend more (LLM assisted) time writing tests.
| gamegoblin wrote:
| I understand the code now, I just didn't know how to
| write a browser plugin and having GPT4 walk me through it
| was 10x faster than reading a tutorial.
| rcme wrote:
| Isn't that pretty much all code?
| czbond wrote:
| Can you explain what you did set up wise for your test? I'm
| following this "space" but the exact, simple pipelines are
| eluding me.
| yacine_ wrote:
| fsreadfilesync and json
| syntaxing wrote:
| Super curious how you did this! Doesn't 30B model require a
| hefty computer to run locally (assuming you're tuning a non-
| quantized version)
| drdaeman wrote:
| Not at all. Even a Raspberry Pi would do - you only need
| ~6GiB RAM for a 4-bit quantized LLaMA model (though it's
| gonna be quite slow). A decent modern desktop machine would
| do just fine, no need for anything extra fancy.
|
| What I'm wondering is how they fed the documents, as all
| those LLMs have limitations on the input sizes.
| jstarfish wrote:
| I've seen reports that it wrecks RPi SD cards in short
| order though, so beware...
|
| > What I'm wondering is how they fed the documents, as all
| those LLMs have limitations on the input sizes.
|
| It's like file hashing at scale, you don't have to read the
| whole stream for every file, just the first 1024/2048 bytes
| (or first few paragraphs).
|
| (This works for classification and sorting, less so for
| summarization.)
| yyyk wrote:
| >~6GiB RAM
|
| That's for the 7B model. The 30B model needs 24GB quantized
| (or 64GB for the unquantized model).
| yacine_ wrote:
| I'm running 30b quant on my 3090. that many 4bits fortunately
| fit into my precious vram
| bob1029 wrote:
| > Sure, semantic text classifiers were possible for a while,
| but making it accessible changes everything.
|
| Binary classification can actually take you all the way in
| terms of classification if you are clever with set theory. It's
| also one of the most traceable & deterministic ways to
| understand how the natural language is being interpreted at
| each step.
|
| The amount of performance required to run something like an SVM
| is laughable compared to what is required to run even baby-tier
| LLMs. If you can reduce the cost of running models to a <1ms
| invocation over a few megabytes of black box, you can easily
| test _thousands_ of these per-user-query. Re-training and
| iterating is much more enjoyable for these reasons. You also
| don 't need any GPUs for this.
|
| At the end of the day, the quality of your data will be the
| biggest issue with older techniques. LLMs can bandaid all sorts
| of weird things that crop up in the real world and aren't
| present in the training data. SVMs cannot tolerate requests
| delivered in the format of Shakespeare (if unexpected). In a
| well-controlled domain, you would probably be able to get away
| with much cheaper options that are also more flexible.
| hospitalJail wrote:
| [dead]
| ticviking wrote:
| The big thing for me and many others is the ability to use the
| tool without sending NDA data to a 3rd party.
|
| The potential amplifying power of _that_ is enormous.
| tyingq wrote:
| Agree, though plain old Bayesian classifiers have been able to
| handle some significant portion of that office work for a long
| time. And not much ever came from it for everyday stuff outside
| of spam filters.
|
| Maybe both the buzz factor and broader applicability means it's
| more likely to happen this go around?
| yacine_ wrote:
| More like: the accessibility is what will make it go around.
| The accessibility is what changes everything! Getting an easy
| to use interface (instructions) over python changes the
| accessibility from the denizens of this website to ~anyone
| with a computer
|
| If you're interested, see this paper that argues that point:
| https://arxiv.org/abs/2302.06541
|
| Essentially, being label efficient is more important than
| being compute efficient, because the biggest computing
| constraint we have is enough humans doing the labeling (and
| knowing how to work a jupyter notebook), not tensor smashing
| nvidia cards
| londons_explore wrote:
| I think the real benefits will be for those with an ad-hoc
| task and no programming/scripting ability.
|
| Sure, you and I know how to write a little script to sort a
| directory of documents into "schoolwork" and "other stuff".
|
| But most people don't have that ability, so giving them that
| would really help accessibility.
| travisjungroth wrote:
| Ease of use is huge, so is deployment. Even as a software
| engineer the overhead for a random classification is so much
| lower.
| xbmcuser wrote:
| Ease of use is what is going to change everything. Using
| natural language to ask something and getting an answer is
| different from what we had before. I knew I could automate a
| lot of my paperwork with scripts but as I am not a programer
| I never gave it more than a cursory look. Last Dec while
| playing around with ChatGPT I was able to get it to write
| some python scripts that resulted in my spending less than
| 20-25 min on tasks that I was spending 20-25 hours on. Now
| could I have written the scripts myself probably but it would
| have taken me months whereas with chat gpt it took me 2-3
| hours to get a working scripts and another 1-2 hours to
| optimise them.
| 1827162 wrote:
| I found this to be _very_ liberating, that I can finally type
| whatever I want into the LLM, without the possibility of the
| government knowing what I am writing. Just being able to do that,
| and have the watchful eye of the state _not_ being able to
| monitor you is amazing.
| seydor wrote:
| you have to check your screen's firmware for that
| ticviking wrote:
| I mean I have moderately high certainty that I'm escaping
| casual surveillance. It's not perfect but it doesn't need to
| be perfect to be good enough.
| FL33TW00D wrote:
| I wrote a similar blog post in Nov 22:
| https://fleetwood.dev/posts/a-case-for-client-side-machine-l...
| Torkel wrote:
| Things are moving so fast in this space I felt like I needed a
| [March] in the title on this one :)
| turnsout wrote:
| Too real-early March even felt different from late March. :D
| croes wrote:
| Local LLMs remove the data protection problem but open the door
| for malicious use on a larger scale.
| colonwqbang wrote:
| It was never possible to keep this technology secret for any
| length of time.
| croes wrote:
| I thought the possibility to run these LLMs on everyday
| hardware would be further on the future and in the beginning
| more limited to big servers.
|
| But this could be the equivalent of the Low Orbit Ion Canon
| for phishers and scammers
| Root_Denied wrote:
| It also makes it easier for the defense/blue team side to
| build counters or monitor and defend the attack surface of
| a given system against these attacks.
|
| Digital arms races are nothing new, this is just the latest
| battlefield.
| sendfoods wrote:
| How realistic is CPU-only inference in the near future?
| travisjungroth wrote:
| It's in the near past. https://github.com/ggerganov/llama.cpp
| abetlen wrote:
| Also worth checking out
| https://github.com/saharNooby/rwkv.cpp which is based on
| Georgi's library and offers support for the RWKV family of
| models which are Apache-2.0 licensed.
| BrutalCoding wrote:
| I've got some of their smaller Raven models running locally
| on my M1 (only 16GB of RAM).
|
| I'm also in the middle of making it user friendly to run
| these models on all platforms (built with Flutter). First
| MacOS release will be out before this weekend:
| https://github.com/BrutalCoding/shady.ai
| [deleted]
| abetlen wrote:
| You can see for yourself (assuming you have the model weights)
| https://github.com/abetlen/llama-cpp-python
|
| I get around ~140 ms per token running a 13B parameter model on
| a thinkpad laptop with a 14 core Intel i7-9750 processor.
| Because it's CPU inference the initial prompt processing takes
| longer than on GPU so total latency is still higher than I'd
| like. I'm working on some caching solutions that should make
| this bareable for things like chat.
| yieldcrv wrote:
| So interesting how you can already tell that is a 3 week old
| epiphany
| simonw wrote:
| "I think we're going to eventually see a demo showing an open
| source model running on an iPhone as well"
|
| I have Kevin Kwok's SheepyT running on my iPhone right now - it
| uses GPT-J, which is an openly licensed LLM by EleutherAI.
|
| https://twitter.com/antimatter15/status/1644456371121954817
| [deleted]
| agumonkey wrote:
| Were transformers used in other contexts ? biochemistry ?
| geometry .. whatever.
| causi wrote:
| LLAMA isn't there and probably never will be, but the possibility
| of running something equivalent to ChatGPT has certainly made me
| reconsider my GPU purchases. I wonder if in the end will it be
| Nvidia's CUDA advantage or AMD's larger amount of memory that
| will end up being more important when we do get it.
| imaurer wrote:
| Tracking repos and resources for running LLMs locally here:
|
| https://github.com/imaurer/awesome-decentralized-llm
| turnsout wrote:
| Also, this continuously-updated spreadsheet is extremely
| helpful:
| https://docs.google.com/spreadsheets/d/1O5KVQW1Hx5ZAkcg8AIRj...
___________________________________________________________________
(page generated 2023-04-11 23:01 UTC)