[HN Gopher] GPU-Accelerated LLM on an Orange Pi
___________________________________________________________________
GPU-Accelerated LLM on an Orange Pi
Author : tosh
Score : 193 points
Date : 2023-08-15 10:30 UTC (12 hours ago)
(HTM) web link (blog.mlc.ai)
(TXT) w3m dump (blog.mlc.ai)
| pjmlp wrote:
| Best part is that they are using TVM.
| tysam_and wrote:
| This was the part that intrigued me. After years of silent
| begging to PyTorch to support ROCm, suddenly models in general
| supporting so many different platforms feels overwhelming, and
| really good.
|
| I'm crediting llama.cpp of all things for being the boost to
| really up the ante on open source model compilation.
|
| Whatever it is, at least, many of these open source things feel
| like they just 'happen', as an eventuality, but in order for
| that to happen it takes a lot of work from a lot of people!
| Really happy to see the dream of this particular kind of
| democratization opening widely! :)
| kvetching wrote:
| I've been thinking of this. It's just fascinating to me to have a
| small device that you can converse with and knows almost
| everything. Perfect for preppers / survivalists. Store it in a
| faraday cage along with a solar generator.
| jacquesm wrote:
| > knows almost everything
|
| It really doesn't. It doesn't even know what it knows and what
| it doesn't know. Without ways to check up on whether what it
| told you is true or not you may well end up in more trouble
| than where you were before.
| mdp2021 wrote:
| What is the current state of "correspondence between fed text
| and output text"? I.e. how much, when fed in the training
| e.g. that "Spain invested 6000$ in Columbus' first voyage",
| LLMs will repeat that notion exactly?
|
| This, without taking into account reasoning and consistence.
| And already this notion that I picked randomly is not without
| issues: dollars how computed? And, it is not difficult to
| state that Columbus reached the Caribbeans in 1492; more
| complex to "decide" the year of the siege of Troy out of the
| many dates proposed.
|
| But already at the simplified level of determined clear
| notions: if LLMs are told that "A is B", and in absence of
| inconsistency in the training corpus, what is the failure
| rate (i.e. then outputting something critically different)?
|
| > _ways to check up_
|
| Some LLMs work as search engines, outputting not just their
| tentative answer but linked references. A reasonably safe
| practice at this stage is to use LLMs that way: ask then use
| the output to check the reference.
| omneity wrote:
| How about a local wikipedia dump, with precalculated
| embeddings? Then you can perform a similarity search first
| and feed the results to the LLM.
|
| It's less likely to hallucinate this way.
| jacquesm wrote:
| That would make a lot more sense. That way at least you
| have a chance to check up on the output, lest your first
| meal of 'Hedysarum alpinum' ends up being your last.
| bravura wrote:
| The key aspect to this being a good solution is actually
| building a corpus representing as much possible reference
| knowledge needed in scenario. The idea that the answer is
| Wikipedia way underestimates the scope.
|
| The Wikipedia patch doesn't make much sense to me.
|
| What percent of the important questions being asked in
| this doomsday scenario actually have their answer in
| Wikipedia?
|
| If 50% of the time you are left trusting raw LLaMa, then
| you don't really have a decent solution.
|
| I do appreciate the sentiment tho that _future_ or
| finetuned LLMS might fit on an RPi or whatever, and be
| good enough.
| hnuser123456 wrote:
| I think the theory is that an LLM can integrate the
| knowledge of Wikipedia and become something greater than
| the sum of its parts by applying reasoning that is
| explained in one article to situations in other topics
| where the reasoning might not be so well explained. Then
| you can ask naive questions in new scenarios where you
| might not have the background knowledge (or simply the
| mental energy) to figure out a right answer to a
| situation on your own but it powers through for you.
| AFAIK current LLMs are not this abstract. If one type of
| reasoning is more often applied in one scenario and
| another type of reasoning is applied to another, they
| don't have any context beyond the words and they know
| what topics humans usually jump to from other given
| topics.
| [deleted]
| yu3zhou4 wrote:
| It would be great to see such project implemented, I wonder
| how good would it perform
| jacquesm wrote:
| Yes, especially having fact checked output of LLMs would
| be a nice step in the right direction. Throwing out the
| hallucinated bits and keeping the good stuff would make
| LLMs a lot more applicable.
| regularfry wrote:
| https://arxiv.org/pdf/2308.04430.pdf is interesting from
| that point of view. They've tackled it from the
| perspective of avoiding copyright content in training,
| but including it in inference but I _think_ it ought to
| mean less hallucination because they also (claim to)
| solve the attribution problem.
| jacquesm wrote:
| Nice one, thank you. Added to my 'read later today' list,
| the abstract looks very interesting.
| eis wrote:
| Isn't that a bit of a holy grail though? If your software
| can fact check the output of LLMs and prevent
| hallucinations then why not use that as the AI to get the
| answers in the first place?
| jacquesm wrote:
| Because you - hopefully - have a check against something
| that is on average of higher quality than the combined
| input of an LLM.
|
| I'm not sure if this can work or not but it would be nice
| to see a trial, you could probably do this by hand if you
| wanted to by breaking up the answer from an LLM into
| factoids and then to check each of those individually,
| and to assign a score to them based on the amount of
| supporting evidence for the factoid. I'd love that as a
| plug-in to a browser too.
| omneity wrote:
| My hypothesis is that including information in the LLM's
| prompt to support its answer changes the task roughly
| from text generation, very hallucination prone, to text
| summarization, or reformulation with some reasoning, and
| this is less likely to hallucinate.
|
| That was my personal experience in general with ChatGPT
| as well as LLaMa1/2.
| jacquesm wrote:
| A friend and colleague of mine just tried this and the
| first results are quite promising.
| mdp2021 wrote:
| > _a local wikipedia dump_
|
| There exists (at least) a project to train and query an LLM
| on local documents: privateGPT -
| https://github.com/imartinez/privateGPT
|
| It should provide links to the the source with the relevant
| content, to check the exact text:
|
| > _You 'll need to wait 20-30 seconds (depending on your
| machine) while the LLM model consumes the prompt and
| prepares the answer. Once done, it will print the answer
| and the 4 sources it used as context from your documents_
|
| You will have noticed, in that first sentence, that it may
| not be practical, especially on an Orange Pi.
| chank wrote:
| this concept already exists and is in practice at many
| companies that require knowledge driven results.
| https://arxiv.org/abs/2005.11401
| regularfry wrote:
| There was another paper out recently that adds to this:
| https://arxiv.org/pdf/2308.04430.pdf. Looks like a more
| flexible approach to document storage, and it outperforms
| retrieval in context.
|
| They trained up their own LLM, but from the text it seems
| like it might be possible to use any LLaMA-style LM
| without retraining. Not sure though, need to give it a
| proper look.
| yieldcrv wrote:
| Yes like a person
| beepbooptheory wrote:
| The energy use alone would probably rule this out for survival,
| no?
| asdgioanionio wrote:
| Depending on an LLM for survival is a good way to end up dead.
|
| English Wikipedia will fit on an SD card. It's more valuable
| and more practical.
| unforeseen9991 wrote:
| "I am confident those red berries you are describing are
| perfectly fine to eat"
| jan_Sate wrote:
| Orange Pi 5 has an NPU. I wonder if it'd be any faster than using
| GPU.
| duchenne wrote:
| How many tokens per second do you think we can get out of this
| 6TFlops NPU?
| brucethemoose2 wrote:
| For prompt ingestion... I dunno.
|
| Unbatched token generation is basically RAM bandwidth
| limited, as the entire model has to be cycled through for
| each token. I bet theoretical performance is similar to the
| GPU, albeit with much lower power consumption.
| [deleted]
| brucethemoose2 wrote:
| Its not supported by TVM yet, but there is support for Qualcomm
| Hexagon.
|
| You can kinda see some of the supported backends gated behind
| flags in the cmake file: https://github.com/mlc-
| ai/relax/blob/mlc/CMakeLists.txt
| adrian_b wrote:
| It might have been more work to convert the model for the
| RK3588 NPU, even if Rockchip provides an SDK and an automated
| conversion tool that should help (the SDK includes a simulator
| for the NPU, so the converted model can be tried on a PC before
| being deployed on a board like Orange Pi):
|
| https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html
| billconan wrote:
| does mlc work for vision models? for example, the doc mentions
| --max-seq-len MAX_ALLOWED_SEQUENCE_LENGTH as a command line
| option. This seems to imply that it only accepts language models?
|
| Also, it doesn't seem to say anything about the input model's
| format? pytorch weights? onnx?
| villgax wrote:
| I'm already getting 1.5tok on Ubuntu running on Android via
| UserLand w/ Llama.cpp(v2-Q4). Don't really see acceleration. If
| anything I need to see my phone do something actually useful at
| let's say 7-10toks
| brucethemoose2 wrote:
| mlc should already be pretty fast on Vulkan
| regularfry wrote:
| Human speech is in the 2-4 tokens per second range, I think
| that's about where my frustration limit is.
| packetlost wrote:
| I had to make a minor modification to the code to get the Rust
| compiler happy, just add a `.as_slice()` when the compilation
| fails. I'll submit a PR if it's not fixed already.
| junrushao1994 wrote:
| Ah please help us by submitting a PR! I noticed the rust build
| failed last night but didn't get a chance to look into it
| binarybana wrote:
| All PR's greatly appreciated! Is this on the Rust TVM bindings?
| sunshine-o wrote:
| I'm surprised we haven't seen dedicated boxes to self host your
| uncensored & private LLM yet.
|
| A bit like you can self host your apps at home on a Umbrel box.
|
| I wonder if the NVIDIA Jetson serie would be the hardware that
| makes the most sense?
| thehumanmeat wrote:
| See tinybox https://tinygrad.org/
| yunohn wrote:
| 15,000$...
| dsalzman wrote:
| The apple lisa was 25k when adjusted for inflation...prices
| are for early adopters
| turnsout wrote:
| Yes, and the Lisa failed, because there just weren't
| enough adopters at that price.
|
| I struggle to see any sizable market for the tinybox, but
| I wish them good luck.
| Damogran6 wrote:
| Lessons learned on the Lisa were applied to the Mac for
| less money to sell to more people. I suspect the same
| thing will happen with their VR/AR/XR headset.
| turnsout wrote:
| They only got a second crack at it because they were
| floating on sales of the Apple II. If the Lisa had been
| their first product, we'd be saying "Apple who?"
|
| The Vision is a whole different universe... with their
| cash and position they could (and may) take 10 cracks at
| it.
| [deleted]
| JimtheCoder wrote:
| Organization run by Geohot and no actual product delivered
| yet...
| dharma1 wrote:
| You can build your own and you don't need 6 GPUs unless
| you're training
| cstejerean wrote:
| $15k for a box with 144 GB GPU RAM is not bad, but I'm not
| clear on how they're going to run that from a single 1600W
| PSU. That would be 6x 24GB GPUs, and I'm pretty sure you'd
| need 2x 1600W PSUs and two separate 15amp circuits to run
| such a thing at home (in the US).
| elorant wrote:
| You can undervolt gpus without loosing that much
| performance.
| jrockway wrote:
| But realistically, the 2x PSU case was a common crypto
| mining setup and yeah, you have your electrician install
| two circuits right next to each other. Or use a 240V PSU.
| chad1n wrote:
| I was thinking that powerful used phones will be extremely
| valuable in the next years, since they are fairly cheap and
| more powerful than these devices.
| Damogran6 wrote:
| Vendors have been pushing hard for trade-in value when new
| phones are purchased to keep perfectly fine phones out of the
| ecosystem.
| brucethemoose2 wrote:
| Jetson is basically like Apple. Theoretically its good, but the
| models with enough RAM are too expensive.
|
| Smartphones aside, little Ryzen 6000 boxes would be OK.
|
| Used DDR5 laptops with a little discrete GPU would be even
| better. I have one with a broken screen that may be dedicated
| to this very task.
| joshvm wrote:
| You can get an Orin with 8GB. Or the 32/64GB AGX module for
| 1.3/2.3k (3k for the dev kit). Not cheap but plenty of ram
| and 60W power target maximum.
|
| You could maybe run something on the 4GB Jetson Nano?
|
| But very slow: https://www.reddit.com/r/LocalLLaMA/comments/1
| 2c7w15/the_poi...
| brucethemoose2 wrote:
| IIRC 32GB of shared RAM is not enough for llama 70B, and
| 8GB is just _barely_ enough for llama 7B... So yeah, that
| value proposition is not good at all.
|
| A 32GB+ ddr5 laptop with a dGPU and some RAM will (IIRC,
| just barely) do llama 70B for far less money and a similar
| TDP.
| dharma1 wrote:
| For inference? Mobile phones (unless you need it always on).
| Kind of works already with 7B weights, will keep getting better
| jdfedgon wrote:
| Nice achievement.
|
| How many users would realistically be able to use it at the same
| time when running on such a device? I am interested in its
| scalability.
| brucethemoose2 wrote:
| If you can run it as a AI Horde worker, and the home usage is
| sporadic, you could definitely support more than one person.
|
| Otherwise ~1.5 tokens/s is definitely the minimum you'd want
| streaming tokens to a single person.
| tysam_and wrote:
| Not many, since it's slow to begin with.
|
| You'll get a log_2-based scaling efficiency with nearly any
| batchsize increase, pending some limitations (memory, etc).
|
| That should be enough at least to roughly sketch it out.
| gmiller123456 wrote:
| "Scalability" and "Single Board Computer" don't really belong
| in the same sentence. That said, today you can get a
| refurbished mini PC with a lot more power, for a lot less money
| than the higher end SBCs. But I didn't see any info on how
| portable this project is to other hardware.
| jacquesm wrote:
| That's a tricky question. You're going to have to multiplex the
| use of the device, but since these are mostly 'ping-pong' style
| uses you can use something called a 'utilization factor' to
| figure out what a reasonable upper bound is where you still get
| an answer to your query in acceptable time. The typical
| mechanism is an input queue with a single worker to use the
| device. The cut-off is when the queue becomes unacceptably
| long, in which case you would have to throw an error or be
| content with waiting (possibly much) longer for your answer.
| This is usually capped by some hard limit on the length of the
| queue (for instance: available memory) or the fact that the
| queue fills up faster than that it can empty even over a
| complete daily cycle. Once that happens you need more hardware.
| dekhn wrote:
| Actually many inference systems instead batch all requests
| within a time period and submit them as a single shot. It
| increases the average latency but handles more requests per
| unit time. (at least, this is my understanding how production
| serving of expensive models that support batching work)
| btbuildem wrote:
| Given the low cost of the setup, I'd expect this to be a
| single-user solution. Maybe something enabling better smart
| home / smart device interactions?
| raphaelj wrote:
| Any idea how the GPU compares to the Nvidia's Jetson series?
| adrian_b wrote:
| It is difficult to find information about the performance of
| the ARM GPUs that can be compared with that of
| NVIDIA/AMD/Intel.
|
| However it seems that the Mali-G610 MC4 is in about the same
| range as the cheaper models of the old Jetson Xavier.
|
| The newer Jetson Orin models have much faster Ampere GPUs with
| between 1024 and 2048 FP32 ALUs. Nevertheless, the various
| Jetson Orin models have a price between 4 times and 13 times
| higher than a SBC with RK3588 and 16 GB DRAM (especially all
| the Orin models with more than 8 GB DRAM are very expensive)
| and the ratio between their prices is much greater than the
| ratio between their performances.
|
| Any small computer with AMD Phoenix offers a much better GPU
| performance per dollar than any NVIDIA Orin. The use of NVIDIA
| Orin is justified only when one needs a device that is
| qualified for an automotive environment.
| raphaelj wrote:
| Thanks for the answer.
|
| > Any small computer with AMD Phoenix offers a much better
| GPU performance per dollar than any NVIDIA Orin.
|
| > The use of NVIDIA Orin is justified only when one needs a
| device that is qualified for an automotive environment.
|
| Nvidia Orin will use significantly less energy though.
| adrian_b wrote:
| > Nvidia Orin will use significantly less energy though.
|
| Not really.
|
| A Ryzen 7 7840U has a GPU with 768 FP32 ALUs @ 2.7 GHz and
| a NPU that can do 10 TOPS and it has a default TDP of 28 W.
|
| The top Jetson AGX Orin models consume up to 60 W or 75 W,
| but they are so expensive that it does not make sense to
| compare them with a computer with 7840U and 32 GB of
| LPDDR5x-7500 that costs 3 times less.
|
| A comparison that makes more sense is with a Jetson Orin NX
| 16GB (still significantly more expensive), which has a GPU
| with 1024 FP32 ALUs @ 0.918 GHz and it has a default TDP of
| 25 W.
|
| For graphics tasks, Jetson Orin NX would be several times
| slower than an AMD Phoenix, due to its low GPU clock
| frequency and much slower CPU cores. The same is true for
| any programs executed on the CPU cores.
|
| On the other hand, for AI inference, Jetson Orin has very
| fast tensor cores, so it can be many times faster than an
| AMD GPU or an ARM GPU, i.e. Jetson Orin NX 16 GB is claimed
| to be able to do 100 TOPS, so if this is the main intended
| application it can be worthwhile. Nevertheless, the
| usefulness of the Jetson Orin models for AI inference is
| diminished by the fact that their price increases very
| steeply when more memory is desired.
| alchemist1e9 wrote:
| Is it possible a pool of these running smaller sized models could
| somehow be cheaply combined into a MOE approach (as supposedly
| GPT-4 is) so create something cheap but higher quality?
___________________________________________________________________
(page generated 2023-08-15 23:01 UTC)