[HN Gopher] GPU-Accelerated LLM on an Orange Pi
       ___________________________________________________________________
        
       GPU-Accelerated LLM on an Orange Pi
        
       Author : tosh
       Score  : 193 points
       Date   : 2023-08-15 10:30 UTC (12 hours ago)
        
 (HTM) web link (blog.mlc.ai)
 (TXT) w3m dump (blog.mlc.ai)
        
       | pjmlp wrote:
       | Best part is that they are using TVM.
        
         | tysam_and wrote:
         | This was the part that intrigued me. After years of silent
         | begging to PyTorch to support ROCm, suddenly models in general
         | supporting so many different platforms feels overwhelming, and
         | really good.
         | 
         | I'm crediting llama.cpp of all things for being the boost to
         | really up the ante on open source model compilation.
         | 
         | Whatever it is, at least, many of these open source things feel
         | like they just 'happen', as an eventuality, but in order for
         | that to happen it takes a lot of work from a lot of people!
         | Really happy to see the dream of this particular kind of
         | democratization opening widely! :)
        
       | kvetching wrote:
       | I've been thinking of this. It's just fascinating to me to have a
       | small device that you can converse with and knows almost
       | everything. Perfect for preppers / survivalists. Store it in a
       | faraday cage along with a solar generator.
        
         | jacquesm wrote:
         | > knows almost everything
         | 
         | It really doesn't. It doesn't even know what it knows and what
         | it doesn't know. Without ways to check up on whether what it
         | told you is true or not you may well end up in more trouble
         | than where you were before.
        
           | mdp2021 wrote:
           | What is the current state of "correspondence between fed text
           | and output text"? I.e. how much, when fed in the training
           | e.g. that "Spain invested 6000$ in Columbus' first voyage",
           | LLMs will repeat that notion exactly?
           | 
           | This, without taking into account reasoning and consistence.
           | And already this notion that I picked randomly is not without
           | issues: dollars how computed? And, it is not difficult to
           | state that Columbus reached the Caribbeans in 1492; more
           | complex to "decide" the year of the siege of Troy out of the
           | many dates proposed.
           | 
           | But already at the simplified level of determined clear
           | notions: if LLMs are told that "A is B", and in absence of
           | inconsistency in the training corpus, what is the failure
           | rate (i.e. then outputting something critically different)?
           | 
           | > _ways to check up_
           | 
           | Some LLMs work as search engines, outputting not just their
           | tentative answer but linked references. A reasonably safe
           | practice at this stage is to use LLMs that way: ask then use
           | the output to check the reference.
        
           | omneity wrote:
           | How about a local wikipedia dump, with precalculated
           | embeddings? Then you can perform a similarity search first
           | and feed the results to the LLM.
           | 
           | It's less likely to hallucinate this way.
        
             | jacquesm wrote:
             | That would make a lot more sense. That way at least you
             | have a chance to check up on the output, lest your first
             | meal of 'Hedysarum alpinum' ends up being your last.
        
               | bravura wrote:
               | The key aspect to this being a good solution is actually
               | building a corpus representing as much possible reference
               | knowledge needed in scenario. The idea that the answer is
               | Wikipedia way underestimates the scope.
               | 
               | The Wikipedia patch doesn't make much sense to me.
               | 
               | What percent of the important questions being asked in
               | this doomsday scenario actually have their answer in
               | Wikipedia?
               | 
               | If 50% of the time you are left trusting raw LLaMa, then
               | you don't really have a decent solution.
               | 
               | I do appreciate the sentiment tho that _future_ or
               | finetuned LLMS might fit on an RPi or whatever, and be
               | good enough.
        
               | hnuser123456 wrote:
               | I think the theory is that an LLM can integrate the
               | knowledge of Wikipedia and become something greater than
               | the sum of its parts by applying reasoning that is
               | explained in one article to situations in other topics
               | where the reasoning might not be so well explained. Then
               | you can ask naive questions in new scenarios where you
               | might not have the background knowledge (or simply the
               | mental energy) to figure out a right answer to a
               | situation on your own but it powers through for you.
               | AFAIK current LLMs are not this abstract. If one type of
               | reasoning is more often applied in one scenario and
               | another type of reasoning is applied to another, they
               | don't have any context beyond the words and they know
               | what topics humans usually jump to from other given
               | topics.
        
             | [deleted]
        
             | yu3zhou4 wrote:
             | It would be great to see such project implemented, I wonder
             | how good would it perform
        
               | jacquesm wrote:
               | Yes, especially having fact checked output of LLMs would
               | be a nice step in the right direction. Throwing out the
               | hallucinated bits and keeping the good stuff would make
               | LLMs a lot more applicable.
        
               | regularfry wrote:
               | https://arxiv.org/pdf/2308.04430.pdf is interesting from
               | that point of view. They've tackled it from the
               | perspective of avoiding copyright content in training,
               | but including it in inference but I _think_ it ought to
               | mean less hallucination because they also (claim to)
               | solve the attribution problem.
        
               | jacquesm wrote:
               | Nice one, thank you. Added to my 'read later today' list,
               | the abstract looks very interesting.
        
               | eis wrote:
               | Isn't that a bit of a holy grail though? If your software
               | can fact check the output of LLMs and prevent
               | hallucinations then why not use that as the AI to get the
               | answers in the first place?
        
               | jacquesm wrote:
               | Because you - hopefully - have a check against something
               | that is on average of higher quality than the combined
               | input of an LLM.
               | 
               | I'm not sure if this can work or not but it would be nice
               | to see a trial, you could probably do this by hand if you
               | wanted to by breaking up the answer from an LLM into
               | factoids and then to check each of those individually,
               | and to assign a score to them based on the amount of
               | supporting evidence for the factoid. I'd love that as a
               | plug-in to a browser too.
        
               | omneity wrote:
               | My hypothesis is that including information in the LLM's
               | prompt to support its answer changes the task roughly
               | from text generation, very hallucination prone, to text
               | summarization, or reformulation with some reasoning, and
               | this is less likely to hallucinate.
               | 
               | That was my personal experience in general with ChatGPT
               | as well as LLaMa1/2.
        
               | jacquesm wrote:
               | A friend and colleague of mine just tried this and the
               | first results are quite promising.
        
             | mdp2021 wrote:
             | > _a local wikipedia dump_
             | 
             | There exists (at least) a project to train and query an LLM
             | on local documents: privateGPT -
             | https://github.com/imartinez/privateGPT
             | 
             | It should provide links to the the source with the relevant
             | content, to check the exact text:
             | 
             | > _You 'll need to wait 20-30 seconds (depending on your
             | machine) while the LLM model consumes the prompt and
             | prepares the answer. Once done, it will print the answer
             | and the 4 sources it used as context from your documents_
             | 
             | You will have noticed, in that first sentence, that it may
             | not be practical, especially on an Orange Pi.
        
             | chank wrote:
             | this concept already exists and is in practice at many
             | companies that require knowledge driven results.
             | https://arxiv.org/abs/2005.11401
        
               | regularfry wrote:
               | There was another paper out recently that adds to this:
               | https://arxiv.org/pdf/2308.04430.pdf. Looks like a more
               | flexible approach to document storage, and it outperforms
               | retrieval in context.
               | 
               | They trained up their own LLM, but from the text it seems
               | like it might be possible to use any LLaMA-style LM
               | without retraining. Not sure though, need to give it a
               | proper look.
        
           | yieldcrv wrote:
           | Yes like a person
        
         | beepbooptheory wrote:
         | The energy use alone would probably rule this out for survival,
         | no?
        
         | asdgioanionio wrote:
         | Depending on an LLM for survival is a good way to end up dead.
         | 
         | English Wikipedia will fit on an SD card. It's more valuable
         | and more practical.
        
           | unforeseen9991 wrote:
           | "I am confident those red berries you are describing are
           | perfectly fine to eat"
        
       | jan_Sate wrote:
       | Orange Pi 5 has an NPU. I wonder if it'd be any faster than using
       | GPU.
        
         | duchenne wrote:
         | How many tokens per second do you think we can get out of this
         | 6TFlops NPU?
        
           | brucethemoose2 wrote:
           | For prompt ingestion... I dunno.
           | 
           | Unbatched token generation is basically RAM bandwidth
           | limited, as the entire model has to be cycled through for
           | each token. I bet theoretical performance is similar to the
           | GPU, albeit with much lower power consumption.
        
         | [deleted]
        
         | brucethemoose2 wrote:
         | Its not supported by TVM yet, but there is support for Qualcomm
         | Hexagon.
         | 
         | You can kinda see some of the supported backends gated behind
         | flags in the cmake file: https://github.com/mlc-
         | ai/relax/blob/mlc/CMakeLists.txt
        
         | adrian_b wrote:
         | It might have been more work to convert the model for the
         | RK3588 NPU, even if Rockchip provides an SDK and an automated
         | conversion tool that should help (the SDK includes a simulator
         | for the NPU, so the converted model can be tried on a PC before
         | being deployed on a board like Orange Pi):
         | 
         | https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html
        
       | billconan wrote:
       | does mlc work for vision models? for example, the doc mentions
       | --max-seq-len MAX_ALLOWED_SEQUENCE_LENGTH as a command line
       | option. This seems to imply that it only accepts language models?
       | 
       | Also, it doesn't seem to say anything about the input model's
       | format? pytorch weights? onnx?
        
       | villgax wrote:
       | I'm already getting 1.5tok on Ubuntu running on Android via
       | UserLand w/ Llama.cpp(v2-Q4). Don't really see acceleration. If
       | anything I need to see my phone do something actually useful at
       | let's say 7-10toks
        
         | brucethemoose2 wrote:
         | mlc should already be pretty fast on Vulkan
        
         | regularfry wrote:
         | Human speech is in the 2-4 tokens per second range, I think
         | that's about where my frustration limit is.
        
       | packetlost wrote:
       | I had to make a minor modification to the code to get the Rust
       | compiler happy, just add a `.as_slice()` when the compilation
       | fails. I'll submit a PR if it's not fixed already.
        
         | junrushao1994 wrote:
         | Ah please help us by submitting a PR! I noticed the rust build
         | failed last night but didn't get a chance to look into it
        
         | binarybana wrote:
         | All PR's greatly appreciated! Is this on the Rust TVM bindings?
        
       | sunshine-o wrote:
       | I'm surprised we haven't seen dedicated boxes to self host your
       | uncensored & private LLM yet.
       | 
       | A bit like you can self host your apps at home on a Umbrel box.
       | 
       | I wonder if the NVIDIA Jetson serie would be the hardware that
       | makes the most sense?
        
         | thehumanmeat wrote:
         | See tinybox https://tinygrad.org/
        
           | yunohn wrote:
           | 15,000$...
        
             | dsalzman wrote:
             | The apple lisa was 25k when adjusted for inflation...prices
             | are for early adopters
        
               | turnsout wrote:
               | Yes, and the Lisa failed, because there just weren't
               | enough adopters at that price.
               | 
               | I struggle to see any sizable market for the tinybox, but
               | I wish them good luck.
        
               | Damogran6 wrote:
               | Lessons learned on the Lisa were applied to the Mac for
               | less money to sell to more people. I suspect the same
               | thing will happen with their VR/AR/XR headset.
        
               | turnsout wrote:
               | They only got a second crack at it because they were
               | floating on sales of the Apple II. If the Lisa had been
               | their first product, we'd be saying "Apple who?"
               | 
               | The Vision is a whole different universe... with their
               | cash and position they could (and may) take 10 cracks at
               | it.
        
             | [deleted]
        
             | JimtheCoder wrote:
             | Organization run by Geohot and no actual product delivered
             | yet...
        
             | dharma1 wrote:
             | You can build your own and you don't need 6 GPUs unless
             | you're training
        
             | cstejerean wrote:
             | $15k for a box with 144 GB GPU RAM is not bad, but I'm not
             | clear on how they're going to run that from a single 1600W
             | PSU. That would be 6x 24GB GPUs, and I'm pretty sure you'd
             | need 2x 1600W PSUs and two separate 15amp circuits to run
             | such a thing at home (in the US).
        
               | elorant wrote:
               | You can undervolt gpus without loosing that much
               | performance.
        
               | jrockway wrote:
               | But realistically, the 2x PSU case was a common crypto
               | mining setup and yeah, you have your electrician install
               | two circuits right next to each other. Or use a 240V PSU.
        
         | chad1n wrote:
         | I was thinking that powerful used phones will be extremely
         | valuable in the next years, since they are fairly cheap and
         | more powerful than these devices.
        
           | Damogran6 wrote:
           | Vendors have been pushing hard for trade-in value when new
           | phones are purchased to keep perfectly fine phones out of the
           | ecosystem.
        
         | brucethemoose2 wrote:
         | Jetson is basically like Apple. Theoretically its good, but the
         | models with enough RAM are too expensive.
         | 
         | Smartphones aside, little Ryzen 6000 boxes would be OK.
         | 
         | Used DDR5 laptops with a little discrete GPU would be even
         | better. I have one with a broken screen that may be dedicated
         | to this very task.
        
           | joshvm wrote:
           | You can get an Orin with 8GB. Or the 32/64GB AGX module for
           | 1.3/2.3k (3k for the dev kit). Not cheap but plenty of ram
           | and 60W power target maximum.
           | 
           | You could maybe run something on the 4GB Jetson Nano?
           | 
           | But very slow: https://www.reddit.com/r/LocalLLaMA/comments/1
           | 2c7w15/the_poi...
        
             | brucethemoose2 wrote:
             | IIRC 32GB of shared RAM is not enough for llama 70B, and
             | 8GB is just _barely_ enough for llama 7B... So yeah, that
             | value proposition is not good at all.
             | 
             | A 32GB+ ddr5 laptop with a dGPU and some RAM will (IIRC,
             | just barely) do llama 70B for far less money and a similar
             | TDP.
        
         | dharma1 wrote:
         | For inference? Mobile phones (unless you need it always on).
         | Kind of works already with 7B weights, will keep getting better
        
       | jdfedgon wrote:
       | Nice achievement.
       | 
       | How many users would realistically be able to use it at the same
       | time when running on such a device? I am interested in its
       | scalability.
        
         | brucethemoose2 wrote:
         | If you can run it as a AI Horde worker, and the home usage is
         | sporadic, you could definitely support more than one person.
         | 
         | Otherwise ~1.5 tokens/s is definitely the minimum you'd want
         | streaming tokens to a single person.
        
         | tysam_and wrote:
         | Not many, since it's slow to begin with.
         | 
         | You'll get a log_2-based scaling efficiency with nearly any
         | batchsize increase, pending some limitations (memory, etc).
         | 
         | That should be enough at least to roughly sketch it out.
        
         | gmiller123456 wrote:
         | "Scalability" and "Single Board Computer" don't really belong
         | in the same sentence. That said, today you can get a
         | refurbished mini PC with a lot more power, for a lot less money
         | than the higher end SBCs. But I didn't see any info on how
         | portable this project is to other hardware.
        
         | jacquesm wrote:
         | That's a tricky question. You're going to have to multiplex the
         | use of the device, but since these are mostly 'ping-pong' style
         | uses you can use something called a 'utilization factor' to
         | figure out what a reasonable upper bound is where you still get
         | an answer to your query in acceptable time. The typical
         | mechanism is an input queue with a single worker to use the
         | device. The cut-off is when the queue becomes unacceptably
         | long, in which case you would have to throw an error or be
         | content with waiting (possibly much) longer for your answer.
         | This is usually capped by some hard limit on the length of the
         | queue (for instance: available memory) or the fact that the
         | queue fills up faster than that it can empty even over a
         | complete daily cycle. Once that happens you need more hardware.
        
           | dekhn wrote:
           | Actually many inference systems instead batch all requests
           | within a time period and submit them as a single shot. It
           | increases the average latency but handles more requests per
           | unit time. (at least, this is my understanding how production
           | serving of expensive models that support batching work)
        
         | btbuildem wrote:
         | Given the low cost of the setup, I'd expect this to be a
         | single-user solution. Maybe something enabling better smart
         | home / smart device interactions?
        
       | raphaelj wrote:
       | Any idea how the GPU compares to the Nvidia's Jetson series?
        
         | adrian_b wrote:
         | It is difficult to find information about the performance of
         | the ARM GPUs that can be compared with that of
         | NVIDIA/AMD/Intel.
         | 
         | However it seems that the Mali-G610 MC4 is in about the same
         | range as the cheaper models of the old Jetson Xavier.
         | 
         | The newer Jetson Orin models have much faster Ampere GPUs with
         | between 1024 and 2048 FP32 ALUs. Nevertheless, the various
         | Jetson Orin models have a price between 4 times and 13 times
         | higher than a SBC with RK3588 and 16 GB DRAM (especially all
         | the Orin models with more than 8 GB DRAM are very expensive)
         | and the ratio between their prices is much greater than the
         | ratio between their performances.
         | 
         | Any small computer with AMD Phoenix offers a much better GPU
         | performance per dollar than any NVIDIA Orin. The use of NVIDIA
         | Orin is justified only when one needs a device that is
         | qualified for an automotive environment.
        
           | raphaelj wrote:
           | Thanks for the answer.
           | 
           | > Any small computer with AMD Phoenix offers a much better
           | GPU performance per dollar than any NVIDIA Orin.
           | 
           | > The use of NVIDIA Orin is justified only when one needs a
           | device that is qualified for an automotive environment.
           | 
           | Nvidia Orin will use significantly less energy though.
        
             | adrian_b wrote:
             | > Nvidia Orin will use significantly less energy though.
             | 
             | Not really.
             | 
             | A Ryzen 7 7840U has a GPU with 768 FP32 ALUs @ 2.7 GHz and
             | a NPU that can do 10 TOPS and it has a default TDP of 28 W.
             | 
             | The top Jetson AGX Orin models consume up to 60 W or 75 W,
             | but they are so expensive that it does not make sense to
             | compare them with a computer with 7840U and 32 GB of
             | LPDDR5x-7500 that costs 3 times less.
             | 
             | A comparison that makes more sense is with a Jetson Orin NX
             | 16GB (still significantly more expensive), which has a GPU
             | with 1024 FP32 ALUs @ 0.918 GHz and it has a default TDP of
             | 25 W.
             | 
             | For graphics tasks, Jetson Orin NX would be several times
             | slower than an AMD Phoenix, due to its low GPU clock
             | frequency and much slower CPU cores. The same is true for
             | any programs executed on the CPU cores.
             | 
             | On the other hand, for AI inference, Jetson Orin has very
             | fast tensor cores, so it can be many times faster than an
             | AMD GPU or an ARM GPU, i.e. Jetson Orin NX 16 GB is claimed
             | to be able to do 100 TOPS, so if this is the main intended
             | application it can be worthwhile. Nevertheless, the
             | usefulness of the Jetson Orin models for AI inference is
             | diminished by the fact that their price increases very
             | steeply when more memory is desired.
        
       | alchemist1e9 wrote:
       | Is it possible a pool of these running smaller sized models could
       | somehow be cheaply combined into a MOE approach (as supposedly
       | GPT-4 is) so create something cheap but higher quality?
        
       ___________________________________________________________________
       (page generated 2023-08-15 23:01 UTC)