[HN Gopher] Llama.cpp: Port of Facebook's LLaMA model in C/C++, ...
___________________________________________________________________
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple
Silicon support
Author : mrtksn
Score : 921 points
Date : 2023-03-10 20:01 UTC (2 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jcq3 wrote:
| Don't know anything about ML can someone can explain me what is
| this hype about?
| lxe wrote:
| You can run your own ChatGPT on your Macbook.
| stagger87 wrote:
| https://en.wikipedia.org/wiki/Machine_learning
| potatoman22 wrote:
| This is an advanced language model that can now run quickly on
| consumer grade hardware. You used to need thousands of dollars
| of GPUs to run a model as sophisticated as this - now it can be
| done on a laptop,
| mattfrommars wrote:
| Wasn't LLaMa official meant to run on consumer grade machine?
| How does this modify the model to make it work.
|
| All of this is confusing.
| MacsHeadroom wrote:
| Yes but it wasn't made to run on a Mac. This project ported
| LLaMA to Apple Silicon so all the macbook users can finally
| play with what the rest of us have had access to for the
| past couple of weeks.
| hot_gril wrote:
| Run meaning run inference, not train, right?
| TOMDM wrote:
| Yes
| Szpadel wrote:
| I tried llama 14b model by using one of online tools (mentioned
| in one of other hn comments, it claimed to use that model, but
| cannot be 100% sure) and I was very disappointed with results. I
| tried to ask it fairly simple question: to write regex validating
| email address, tried this 2 times and got responses: "what is
| your question" and "please wait..." so it just doged response. in
| contrast chatgpt was able to provide me with simple regex and
| also RFC compliant version when asked.
| lxe wrote:
| I've gotten great q&a / conversational results from llama 7b
| and 13b using https://github.com/oobabooga/text-generation-
| webui. ], [ "How does the
| Schrodinger's cat thought experiment relate to the principles
| of quantum mechanics?", "In this famous paradox, a
| cat inside a box is simultaneously both alive and dead due to
| the nature of subatomic particles. This demonstrates that at
| the smallest scales of reality, we cannot predict whether
| something will happen until we observe it happening."
| ], [ "What are some of the potential
| consequences of climate change on global food security?",
| "According to researchers at Stanford University, rising
| temperatures could lead to crop failures and reduced harvests
| for staple crops such as rice, wheat, corn, and soybeans by
| midcentury." ], [ "Can you provide a
| brief overview of the history of political philosophy, from
| Ancient Greece to the present day?", "Absolutely!
| Political philosophy is a branch of philosophy concerning the
| role of government in society. In ancient times, Plato and
| Aristotle were among the first thinkers to consider the subject
| in depth. Since then, other influential philosophers such as
| Thomas Hobbes and John Locke have expanded upon these ideas.
| Today, contemporary philosophers such as John Rawls continue to
| debate the merits of different political systems." ]
| lxe wrote:
| And 30b: Give me a basic recipe for easy
| spaghetti carbonara Here is a basic recipe for
| spaghetti carbonara: Ingredients: 1 lb
| spaghetti, 3 tbsp butter, 2 tbsp olive oil, 1/4 cup grated
| Parmesan cheese, 1 egg, 1/2 teaspoon salt, 1/4 teaspoon
| pepper, 1/2 cup chopped parsley, 1/2 pound bacon, 1 clove
| garlic, 1/4 cup heavy cream. Directions: Cook
| spaghetti according to package directions; drain. Heat butter
| and oil in large skillet on medium heat. Add bacon; cook 4
| minutes or until crisp. Remove from skillet. Add garlic to
| drippings in pan; saute 1 minute. Stir in egg, salt, pepper,
| and parsley; stirring constantly. Add bacon and spaghetti;
| toss well. Sprinkle with cheese. Serves 4.
| zabzonk wrote:
| doesn't say what to do with cream. sprinkling with cheese
| is a bit wrong. carbonara sauce is more difficult to do
| than you (or an AI) might think.
| zabzonk wrote:
| well, does it say what to do with cream?
| underlines wrote:
| cream stays far far away from carbonara. otherwise it's
| not carbonara.
|
| carbonara sauce is simply pecorino or parmigiano cheese
| mixed with eggs or just yolks and pepper and guanciale or
| pancetta.
|
| NO CREAM, NO MILK, NO HAM, NO BACON! basta! /endofrant
| vidarh wrote:
| Probably because classic Carbonara does not include
| cream. Including cheese is normal on the other hand.
|
| So, yeah, it's not perfect but it's close to being
| reasonable.
| IanCal wrote:
| Outside of it being possibly a smaller model, the prompts
| should be different as llama hasn't been trained to take
| instructions so it would I think need to be framed more as "A
| regex for emails is " and let it complete the rest. Just in
| case you have tried the exact same prompt.
|
| Also those messages sound suspiciously like error messages from
| a system around it.
| endymi0n wrote:
| Llama is neither instruction finetuned nor human reinforcement
| learning ranked. It's a base model like the original GPT3 that
| needs much more careful prompting and few-shot reasoning to
| produce similar results.
| dmm wrote:
| sillysaurusx made some posts about tweaks that greatly improved
| output for him. Maybe they would help you:
|
| """
|
| Fixed the llama sampler. After turning off top_p, adding top_k
| 40, setting temp to 0.7, and adding a repetition penalty of
| 1/0.85, llama 7B is looking nice.
|
| """
|
| https://twitter.com/theshawwn/status/1632569215348531201?cxt...
|
| https://twitter.com/theshawwn/status/1633456289639542789?cxt...
| Szpadel wrote:
| for reference I tried or here: https://lite.koboldai.net/
|
| with chat mode, and parameters: temp 0.62, top_p 0.9,
| repetition penalty 1.08
|
| is free to try, but then you have to wait in queue, therefore
| I had to wait 10min per response and I didn't had patience to
| test it further
| random_kris wrote:
| Can someone create a wasm with typescript so that i can run this
| in the browser
| kossTKR wrote:
| how long before someone creates a simple GUI for this?
|
| That + a small bit of optimisation and everyone with a newer Mac
| / iPhone will be able to run something akin to chatGPT locally!
|
| Isn't this a pretty crazy development - just weeks ago people
| said this would be impossible.
|
| From this thread the 13b model runs just as fast as chatGPT on a
| M2 Macbook Air, and it's not even using the Neural Engine yet so
| will become significantly faster once that is utilised - wow!
| MacsHeadroom wrote:
| People have been running LLaMA in 4bit quickly on cheap
| hardware with a simple GUI for over a week using
| https://github.com/oobabooga/text-generation-webui
|
| Just not on Macs. (that repo does not support Apple Silicon)
| jjkeddo199 wrote:
| Insanity! This is the same guy who wrote Whisper C++! How does he
| do this? I feel like I am a side character in some cartoon
| gasping at the unthinkable power level of the main character.
| Alifatisk wrote:
| What's even more impressive is this part
|
| > This was hacked in an evening
| DesiLurker wrote:
| but why would you do C++ when its quite clear ML load is highly
| parallel. the page says vectorized by NEON but no mention whether
| its autovectorized by gcc or hand optimized. That will have a
| pretty significant performance impact.
| tempaccount420 wrote:
| You can read the code. It's open source.
| eigenvalue wrote:
| Seems like it's extremely high performance for me on an M1
| running the 7B model. Totally usable.
| DesiLurker wrote:
| just because something is high performance does not means it
| cannot be improved by say another 2X. My point was not
| whether its usable or not, it is that if you are going to run
| on CPU vectorization is rather important part and its odd
| that the landing page has no remark about it.
| graderjs wrote:
| ggerganov is a genius! I'm using his excellent whisper.cpp in my
| WisprNote offline private transcribing MacOS app. It 10x the
| speed of the regular models. He even invented a custom format for
| the models! The work this guy is doing--first class or beyond!!!
| eigenvalue wrote:
| This is so awesome and exciting. I have an M1 iMac and it was
| trivially easy to get this working and generating text. And the
| performance is VERY impressive, especially considering that it's
| not even using any of the built in "neural compute" stuff. Also,
| the model seems like it doesn't have any political correctness
| conditioning based on some of the completions it has given me on
| controversial prompts. I can't wait until someone gets the 13b
| model working (sounds like this should happen in the next day or
| so) and gets the repetition penalty working.
| sillysaurusx wrote:
| Repetition penalty is a matter of, generate a token, then
| multiply that logit by the penalty. (If the logit is negative,
| divide instead of multiply.)
|
| https://github.com/shawwn/llama has an implementation (check
| the commit history).
| lynguist wrote:
| It is using the built-in neural accelerators, that's why it's
| fast, that's why it's only supported on Macs so far. The code
| makes use of official Apple APIs which delegate the necessary
| BLAS calls to the available hardware.
| kamranjon wrote:
| Confusingly there are 2 mechanisms to do matrix operations on
| the new apple hardware - AMX (https://github.com/corsix/amx)
| - and the ANE (apple neural engine) - which is enabled by
| CoreML. This code does not run on the neural engine but the
| author has a branch for his whisper.cpp project which uses it
| here: https://github.com/ggerganov/whisper.cpp/pull/566 - so
| it may not be long before we see it applied here as well. All
| of this is to say that it actually could get significantly
| faster if some of this work was able to be handed to the ANE
| with CoreML.
| lynguist wrote:
| You're right! I wrote it too fast without thinking!
| raphlinus wrote:
| Three. You can also do it in Metal, which as of recently
| has cooperative matrix multiplication in the form of the
| simd_matrix type (this is similar functionality as "tensor
| cores" in the Nvidia world). I have no idea what the
| software support is, but I have seen analysis suggesting
| that the raw tensor multiplication throughput is larger
| than ANE for the high-end GPUs.
| simonw wrote:
| I got this working on my 64GB M2 MacBook Pro!
|
| Wrote detailed notes here for anyone else who wants to try this:
| https://til.simonwillison.net/llms/llama-7b-m2
| ls612 wrote:
| I have the 64GB M2 Max I will have to try this out, its amazing
| that this laptop can do things my high end desktop can't do,
| Apple really made some good architectural choices with apple
| silicon.
| nvegater wrote:
| Amazing. Thank you very much !
| oidar wrote:
| Thank you!
| jeffhwang wrote:
| Thank you for these helpful notes!!
| xiphias2 wrote:
| George Hotz already implemented LLaMA 7B and 15B on Twitch
| yesterday on GPU in Tunygrad llama branch:
|
| https://github.com/geohot/tinygrad/tree/llama
|
| The only problem is that it's swapping on 16GB Macbook, so you
| need at least 24GB in practice.
| diimdeep wrote:
| There are two coding sessions streamed so far, very interesting
| to watch
|
| George Hotz | Programming | can we fit a LLaMA inside a
| tinygrad? https://www.youtube.com/watch?v=0kRDs9BW2NU
|
| George Hotz | Programming | ChatLLaMA: get in losers we're
| building a chatbot https://www.youtube.com/watch?v=nctqc8FBJ2U
| jalino23 wrote:
| that is so impressive!
| attractivechaos wrote:
| Both are _very_ impressive. A nice thing about Gerganov 's
| implementation is that it is written in almost pure C. Arguably
| easier for deployment.
| AlecSchueler wrote:
| What does almost mean in this case?
| [deleted]
| bigstumpy wrote:
| There is also C++
| codetrotter wrote:
| Iow it probably wouldn't compile with an actual C only
| compiler, but by and large it looks more like C than like
| C++?
| lostmsu wrote:
| It's not tinygrad really, it is PyTorch.
| capableweb wrote:
| Unfamiliar with the domain, but is it? Looking through the
| changes
| (https://github.com/geohot/tinygrad/compare/master...llama)
| it seems to not mention pytorch at all but tinygrad + numpy.
| BoberMod wrote:
| There is also a gpu-acelerated fork of the original repo
|
| https://github.com/remixer-dec/llama-mps
| xiphias2 wrote:
| > For 7B model, it always goes above 32gb of RAM,
|
| That's double of what Tinygrad uses
| DennisAleynikov wrote:
| Tinygrad is using openCL right?
| apetresc wrote:
| Can you link to the Twitch stream VoD?
| vopi wrote:
| https://www.twitch.tv/georgehotz/videos?filter=archives&sort.
| ..
|
| although, there is a VOD channel on YT that might be better.
| yieldcrv wrote:
| why does it say video unavailable?
| vopi wrote:
| Hmm. I'm not sure. Presumably this will be updated soon:
| https://www.youtube.com/@geohotarchive/videos
| stagger87 wrote:
| I'm pretty sure he only makes his past twitch streams
| available to subscribers. It's weird they are available
| on Youtube, maybe he doesn't know that's something he can
| change.
| jonas21 wrote:
| I don't think he runs the YouTube channel - it's managed
| by some of his fans. But like what's he gonna do about
| it? Send them a takedown notice?
| botdan wrote:
| Not sure about his more recent content but he used to
| have a policy that the VODs could be uploaded in full as
| long as they were uncut and not modified. If you go watch
| some older VODs he says so at the beginning.
| Mistletoe wrote:
| This is such a refreshing and neat way to use Twitch.
| 999900000999 wrote:
| How long until someone puts out a docker image of this?
|
| Ideally with a nice rest API, I can't imagine it's too hard to
| do.
| recuter wrote:
| > Currently, only LLaMA-7B is supported since I haven't figured
| out how to merge the tensors of the bigger models. However, in
| theory, you should be able to run 65B on a 64GB MacBook
|
| Suddenly the choices Apple made with its silicon are looking like
| pure genius as there will be a lot of apps using this that are
| essentially exclusive to their platform. Even with the egregious
| ram pricing.
|
| With a lot of fine tuning if you squint this is a
| useful/convincing "ChatGPT on a laptop" minus the corporate
| lobotomy only a few short months after release. Very exciting! I
| actually care about the upcoming Mac Pro now.
|
| $3999 Mac Studio with 64GB ram. +$800 for 128GB.
| [deleted]
| londons_explore wrote:
| LLaMA doesn't perform very well with answering questions.
|
| If you ask it "What color is the sky?" It will reply with
| something like "Why is ice cold? Why do we exist?"
| RoboTeddy wrote:
| LLaMA isn't built on RLHF, so it may be necessary to create a
| more extensive prompt. For example:
|
| ```
|
| You are a super intelligent honest question-answering system.
|
| Q: What's 2+2?
|
| A: 4
|
| Q: What color is the sky?
|
| A:
|
| ```
| guestbest wrote:
| This is Commodore 64 tier answers
| circuit10 wrote:
| That's just an example
| lgas wrote:
| That's the point, even C64 tier examples in one shot or
| few shot learning do wonders for changing the behavior of
| the model.
| rastapasta42 wrote:
| Ask dumb questions receive dumb answers.
| nabakin wrote:
| It performs very well, but you have to give it the right
| prompt and model params. I imagine it will be ChatGPT level
| once it is trained with RLHF
| Nevermark wrote:
| Given that the RAM is directly accessible by CPU, GPU and
| DPU/Neural cores, it really is premium RAM.
|
| Apple's visionary hardware team has finally caught up with
| Apple's visionary high RAM prices! :)
| DennisAleynikov wrote:
| Also um... Idk if anyone cares but apple didn't do anything
| to make unified memory.
|
| All arm chips do this lmao
|
| Apple just has the best known arm chips with the highest
| mobile performance (yes faster server arm chips exist too)
| fooker wrote:
| It's amazing how Apple "doesn't do" anything, but manages
| to define industry trends over and over again!
| 7to2 wrote:
| That's what brand recognition and brand loyalty (and the
| money they generate) allows you to do. This is not
| necessarily a bad thing since it forces other competitors
| - who like to artificially limit their products - to
| actually get of their greedy asses and compete.
| alphanullmeric wrote:
| Wonder why AMD/Intel/Nvidia haven't invented some sort of
| device that allows the processor and graphics to share memory
| like Apple has done.
| chucklenorris wrote:
| Games don't want slow memory accesses because it tanks the
| fps so there was no incentive to have a working
| implementation (except in laptops).
|
| And also - vram is a moat that keeps the cost of
| "professional" cards absurdly high without making them
| actually faster than consumer cards.
| shaklee3 wrote:
| as pointed out, both Jetson and UM do this, but the upcoming
| Grace does it at higher bandwidth than the apple chip.
| mhh__ wrote:
| 1. They have
|
| 2. CPUs and GPUs typically disagree on whether they want high
| bandwidth or low latency, apple managed to keep both happy
| but it's very hard to do on a PC where the RAM,CPU, and GPU
| are quite far apart and also nowhere near as homogenous as
| Apple have them.
| slaymaker1907 wrote:
| Nvidia actually does support this via unified memory. It's
| actually an amazing performance trick since you can avoid
| launching a bunch of kernels just to ferry data back and
| forth. I did this on a a GTX 1080 for a school project to
| speed up random shuffling of large amounts of data.
|
| However, even without this feature, you can implement this
| sort of thing manually in most cases and you're already being
| careful on a GPU to respect the cache (only working with one
| contiguous set of data of memory at a time).
|
| Really, we just need some good systems devs working on
| running these huge models.
| dagmx wrote:
| Both AMD and Intel have APUs. However they're limited by
| being minority products (so very low support from software)
| and often have limits on how much ram can be accessed by the
| GPU and/or have very weak GPUs.
| PragmaticPulp wrote:
| PCIe devices like GPUs can access system memory.
|
| Integrated GPUs also access system memory via the same bus as
| the CPU.
|
| It's not really a new technique. Apple just shipped a highly
| integrated unit with large memory bandwidth.
| recuter wrote:
| That's a pretty important just. They chose to go down this
| path at this time and shipped, now I can run the 64B Llama
| on a widely available $3,999 prosumer device.
|
| How much would a PC that can do that currently cost me and
| can I have it by tomorrow?
| smoldesu wrote:
| _puts on my jaded realist hat_
|
| It's a great option if you have the hardware and want the
| speed. It's table stakes when other vendors like Nvidia,
| Intel, Qualcomm and Microsoft have acceleration though.
| Raw-compute-for-the-buck has always been a blowout with
| Apple Silicon GPUs, and it's not any prettier now that
| 4nm 40XX series cards are available. Hell, an Intel A770
| with 16gb of VRAM is still cheaper than adding 16gb of
| RAM to a Mac Mini.
|
| It's good stuff, but pitched a bit hard with all the
| marketing. From where I'm standing, it looks like Apple
| is playing catch-up with their GPUs and acceleration
| tech.
|
| _tentatively removes hat of jaded realism_
| KVFinn wrote:
| >How much would a PC that can do that currently cost me
| and can I have it by tomorrow?
|
| At the moment, seems like Apple has an edge here. On PC
| for single GPU you need an NVIDIA A40, which used prices
| for is about $2500, and not at retail stores.
|
| If you don't mind having two GPUs then two $800 3090 GPUs
| works, but that's a workstation build you'll have to
| order from Puget or something. That's probably faster
| than Apple.
|
| My gut instinct is that there's some low hanging fruit
| here and in the next couple weeks 64B Llama will run
| comparably or faster on any PC with a single 4090/3090
| and 64 or 128 GB of system memory. But probably not any
| PC laptops that aren't 17 inch beasts, Apple will keep
| that advantage.
| brookst wrote:
| ...and for models that require 64GB of VRAM? 120GB of
| VRAM?
|
| You can get a 128GB UMA mac for less than a single 48GB
| a100, let alone a single 96GB a100.
|
| I think Apple got incredibly lucky here, but I don't see
| how the PC world catches them any time soon. We've all
| known that UMA is theoretically better for ages, but
| Apple's timing couldn't be better. And scale economies
| mean they can sell the same chip to people who need 100GB
| of system RAM and people who need 100GB of VRAM.
|
| If they can get their GPU / neural performance up and
| sort out their terrible relationship with academic
| research, they could snipe ML away from nvidia. It seems
| very unlikely, but it's kind of stunning that it's even
| in the realm of possibility.
| smoldesu wrote:
| > they could snipe ML away from nvidia.
|
| If Nvidia announced tomorrow that they were cancelling
| every datacenter deal they had, open-sourcing CUDA and
| publishing their entire patent library to the creative
| commons, I would still not believe you.
|
| This is a fun project for people with Apple Silicon
| machines who want to participate in the AI happenings,
| but I don't think you can warp it into a call for
| Nvidia's head. Let's wait until Apple pulls the curtains
| on their rackmount Mac Pros, so we can compare it with
| Nvidia's ARM server offerings: https://www.nvidia.com/en-
| us/data-center/grace-cpu/
| jokethrowaway wrote:
| You can buy a prebuilt pc with a 4090 for less - which is
| significantly more powerful but still in the 3xxx$.
|
| You could go cheaper with a 3090 which has the same vram
| and it's just slower.
|
| I think the best combo is a serious Nvidia pc for AI + a
| cheap MacBook air for portability.
| jokethrowaway wrote:
| There's even an Asus laptop with a 4099 for 3999$
| fomine3 wrote:
| Laptop 4090 is desktop 4080
| MacsHeadroom wrote:
| You need two 3090s or 4090s to fit 65B even in 4bit. It's
| a big one.
|
| That said, if you're fine with slower speeds then two
| P40s could get the job done for $150 each. (Not sure how
| much slower this would go though.)
| kkielhofner wrote:
| The Nvidia Jetson/Tegra line did (and does) unified memory.
| Released 2014[0]. Nvidia already has ARM cores. They're just
| terrible compared to Apple, Samsung, etc.
|
| This is what the attempted ARM acquisition by Nvidia was
| about - with the ARM talent, IP, etc they'd be able to
| integrate more than just memory (GPU, CPU, connectivity via
| Mellanox, etc).
|
| Regulators shut it down (for good reason) but I can't help
| but think we would have seen some really interesting and
| revolutionary platforms come from it.
|
| [0] - https://en.m.wikipedia.org/wiki/Tegra#Tegra_K1
| userbinator wrote:
| Intel GPUs have had that feature for over two decades, and it
| was also called UMA; synonymous with cheap and slow, before
| Apple hyped that term and made a UMA system that actually had
| decent performance.
| fulafel wrote:
| They have, see eg this from 10 years ago:
| https://www.tomshardware.com/news/AMD-HSA-hUMA-
| APU,22324.htm...
|
| Implementing the software support and getting operating
| systems to play along and fragmentation between GPU vendors,
| as always with GPUs on x86, have been the problems. From all
| accounts it's been working reasonably well on the consoles
| though.
|
| Also chicken-and-egg: low GPU compute usage uptake outside of
| games has meant it's not improved lately.
| technerder wrote:
| Absolutely love ggerganov's approach with models like this and
| Whisper. It's just awesome being able to experiment with (what I
| consider) complex models without needing a billion python/c/cpp
| dependencies!
| 0xakhil wrote:
| Does Llama model has a token limit like ChatGPT api (4k tokens)?
| MacsHeadroom wrote:
| It's currently 2048 tokens. But like GPT-3, that can be
| improved in the future.
| hnfong wrote:
| I assumed I needed a M1 pro or better due to RAM requirements,
| but I was able to run the 7B model on a 16GB M1 Mac Mini. The
| system reported 4GB RAM usage for the inference process. I
| suspect it's possible to run the 13B model as well.
| mrtksn wrote:
| You can, I was able to run 13B on my 16GB 8c8g M1 Air. The
| performance was 2-3 tokens/second. It felt on par with ChatGPT
| on a busy day.
| 0xbadc0de5 wrote:
| That's all fine and good. But to do anything useful, you're going
| to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much
| VRAM as possible. Unlike the diffusion models, LLM's are very
| memory-intensive, even at 4-bit GPTQ. The larger models like
| llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU.
| The llama-65b-4bit should run on a dual 3090/4090 rig.
|
| Coupled with the leaked Bing prompt and text-generation-webui,
| the results are quite impressive.
| liuliu wrote:
| Here is some quick math: these devices has SSD read speed
| somewhere around 2GiB/s. With 4-bit quantization, we are
| looking at loading 4B parameters per second. That means we need
| 8s per token for 30B model. Hmmm, the math is a bit off (or I
| need to look closer whether we can do more tokens per iteration
| with some batching).
| visarga wrote:
| You can use Speculative Sampling, where a draft model is used
| to generate short continuations of a sequence, which are
| scored in parallel by the large model. The draft model can be
| small, and you only need to call the large model from time to
| time, so you can stream it from SSD or cheap RAM.
|
| Using LRDIMM DDR4 at the price of less than $1000 it is
| possible to stream GPT-3 five times a second, in 4bit
| quantisation. Multiply that with the 2-2.5x speedup from
| Speculative Sampling.
|
| >Accelerating Large Language Model Decoding with Speculative
| Sampling
|
| https://arxiv.org/abs/2302.01318
| 0xbadc0de5 wrote:
| I think your math is missing some details.
|
| An RTX 4090 has a memory bandwidth of 1,008 GB/s.
|
| PCIe 4 x16 has a 32GB/s bandwidth.
|
| DDR4 RAM has 3200 MT/s transfer rate.
|
| An AMD 5900x can easily max that out.
|
| A good NVME can hit 7 GB/s read speeds or better.
|
| And the 4-bit CUDA kernels can pack 16x 4-bit ints into a
| single 64-bit transfer / register.
| brookst wrote:
| Macs have UMA so an off the shelf Mac can use up to about 120GB
| of vram. Far more than any consumer card, more than the biggest
| a100.
|
| GPU power is lower, of course, but pure vram is not a problem.
| kiratp wrote:
| Apple Silicon uses unified memory so laptops have up to 64GB of
| VRAM and the Max sitio can have up to 128 GB.
|
| That makes them uniquely "powerful" for inference with large
| models.
| lynguist wrote:
| Nowadays up to 96GB on the laptops.
| stu2b50 wrote:
| VRAM is the thing that Apple Silicon is going to have in excess
| compared to anything even close in price. MacBook Airs can have
| 14-15GB of VRAM if necessary.
| kir-gadjello wrote:
| RAM read bandwidth of M1/M2 is still not really competitive
| with the large GPUs like RTX3090, but it's getting close,
| compared with the usual CPU setups.
| gorbypark wrote:
| Is there an upper limit on the usable VRAM on Airs, or is it
| just tied to RAM size minus a bit for the OS? Just got a 24GB
| Air M2 and your comment made me wonder if it was capped at n
| amount of VRAM, or if it's just that Air's now can have 24GB
| of ram compared to 16GB with the M1 Airs?
| 2bitencryption wrote:
| Could someone with experience explain: what's the theoretical
| minimum hardware requirement for llama 7B, 15B, etc, that still
| provides output on the order of <1sec/token?
|
| It seems like we can pull some tricks, like using F16, and some
| kind of quantization, etc.
|
| At the end of the day, how much overhead is left that can be
| reduced? What can I expect to have running on 16gb ram with a
| 3080 and a midrange AMD processor?
| loufe wrote:
| At 4 bits the 13B LLaMa model can run on a 10GB card!
| thewataccount wrote:
| 16GB of vram can run the 7B for sure, I'm not sure what the
| most cutting-edge memory optimization but the 15B is going to
| be pretty tight I'm not sure that'll fit with what I know of at
| least, I've got it working at a bit over 20gb of vram I think
| at 8bit.
|
| If you can't fit it all in vram you can still run it but it'll
| be slooooow, at least that's been my experience with the 30b.
| 0xbadc0de5 wrote:
| The 4-bit GPTQ LLaMA models are the current top-performers.
| This site has done a lot of the heavy lifting:
| https://github.com/qwopqwop200/GPTQ-for-LLaMa
|
| With 30b-4bit on a RTX 4090, I'm seeing numbers like:
|
| Output generated in 4.17 seconds (4.03 tokens/s, 21 tokens)
|
| Output generated in 4.38 seconds (4.25 tokens/s, 23 tokens)
|
| Output generated in 4.57 seconds (4.25 tokens/s, 24 tokens)
|
| Output generated in 3.86 seconds (3.40 tokens/s, 17 tokens)
|
| The lower size (7b, 13b) are even faster with lower memory use.
| A 16GB 3080 should be able to run the 13b at 4-bit just fine
| with reasonable (>1 token/s) latency.
| yieldcrv wrote:
| And on an M1?
|
| I have 64gb available
| [deleted]
| nabakin wrote:
| With a recent PR, text-generation-webui makes this very easy
| to use https://github.com/oobabooga/text-generation-
| webui/pull/206
| TaylorAlexander wrote:
| Well I was able to run the original code with the 7B model on
| 16GB vram: https://news.ycombinator.com/item?id=35013604
|
| The output I got was underwhelming, though I did not attempt
| any tuning.
| fdb wrote:
| The author just made an update that makes the generation much
| better, even with the 7B model:
|
| https://twitter.com/ggerganov/status/1634310199170179075
|
| I tried it out myself (git pull && make) and the difference
| in results are day and night! It's amazing to play with,
| although you should prompt it differently than ChatGPT (more
| like the GPT-3 API).
| fnbr wrote:
| parameter tuning is pretty necessary, according to anecdotes.
| People on twitter have got good results by changing the
| default parameters.
| DeathArrow wrote:
| How does one learn to do stuff like this? You first learn doing
| AI algorithms in Python and then transfer the knowledge to C++ or
| you learn doing them in C++ from the start?
| PartiallyTyped wrote:
| You first need the basic building blocks which is matrix
| multiplication. You may use libraries here.
|
| Then it's a matter of loading the model correctly.
|
| If you are just reimplementing and not doing research you don't
| need the mathematics.
|
| The algorithm is rather simple too. The issue is more about
| engineering than machine learning.
| sgt wrote:
| Does anyone know how many languages this supports? I know that FB
| has been translated to a ton of languages. Will those
| translations benefit the models in LLaMA? Or am I
| misunderstanding the point of this?
| MacsHeadroom wrote:
| The point of it is to generate arbitrary text locally, like the
| GPT-3 API does remotely.
|
| It can "say"/talk about anything an average IQ person with
| knowledge of the entire internet and most books in existence
| could. So if you prompt it to write 100 pages on the
| differences between positive and negative law, as a poem, while
| never using a word with the letter "F" it will spit that out
| for you without any issue.
|
| It can also program quite well, create recipes, debate with
| you, impersonate anyone, and lots more. And it does all of this
| offline, in airplane mode, locally on your PC or Mac.
|
| It's good at translation but is probably one of the least
| efficient ways to translate text when models specifically for
| translation exist.
| sgt wrote:
| What models exist for translations? The reason I wonder is
| that a lot of community work has gone into producing high
| quality translations of Facebook. I have actually submitted a
| few translations myself. If Meta could release some of those
| models, it would be very powerful.
|
| Right now I see "google translate" type quality everywhere
| and it's pretty bad, since there are often sentences you
| can't translate unless the technology understands the context
| and meaning.
| lxe wrote:
| I'm running 4-bit quantized llamas on torch/cuda with
| https://github.com/qwopqwop200/GPTQ-for-LLaMa, and I'm seeing
| significant tokens/second perf degradation compared to 8-bit
| bitsandbytes mode. I'm very new to this, and understand very
| little detail, but I thought it would be faster?
| washadjeffmad wrote:
| In case anyone catches this late (and anything older than a few
| hours in AI/ML is!), some of the original llama HF weights were
| not built correctly and gave poor output.
|
| Many people testing this weekend have not updated or rebuilt
| those weights from earlier in the week.
| sillysaurusx wrote:
| Eh, I'd expect it to be slower if anything. Think about it like
| this. If you write an image renderer, bitmap would be the
| fastest, because it's already decompressed. 4-bit quantization
| is a compression algorithm.
|
| It depends on the details of memory bandwidth vs compute
| though.
| nl wrote:
| I'm getting 56.38 ms per token on my 32GB M1 Max using this code
| on the 7GB model.
|
| Very usable!
| 2Gkashmiri wrote:
| what model are you using?
|
| edit: i mean 6B, 13b or 30B?
| garblegarble wrote:
| It'll be 7B they're referring to, on my M1 Max 32GB with a
| 4000 token output request I get 67ms/token on 7B (4bit) and
| 154ms/token on 13B (4bit)... I've made a tweak to the code to
| increase the context size but it doesn't seem to change perf.
| main: mem per token = 22357508 bytes main: load
| time = 2741.67 ms main: sample time = 156.68 ms
| main: predict time = 11399.12 ms / 154.04 ms per token
| main: total time = 14914.39 ms
| nl wrote:
| This was generating 2000 tokens, so it seems to get
| slightly faster on longer generation runs maybe?
| dekhn wrote:
| This is sort of the polar opposite of how modern high performance
| ML frameworks are built. Skimming the code, there's a ton of
| boilerplate for the various operations that could be library-ized
| and generified, if that makes sense.
|
| I actually really like minimal implementations of state-of-the-
| art systems because the code is much easier to understand (modern
| frameworks are super-complex) but I wonder what it means long-
| term if you don't need frameworks.
| SCLeo wrote:
| This is just inference. The core at most ML library is the auto
| differentiation capability. It will be extremely tedious if you
| are to calculate the gradients manually. Or, if you implemented
| your own AD, then it is effectively a minified version of a ML
| library.
| version_five wrote:
| llama.cpp/ggml.h // GGML Tensor Library
| ... // This library implements: // - a set
| of tensor operations // - automatic differentiation
| // - basic optimization algorithms
| mhh__ wrote:
| > autodiff
|
| There's a lot of // TODO: implement
| backward
|
| in there
| version_five wrote:
| Ok thanks for digging deeper, I didn't realize that, and
| obviously that invalidates the excerpt I posted above
| ojosilva wrote:
| Interesting. But how about the Apple Neural Engine (ANE)? I've
| always wondered if the ANE is ML worthy, maybe it's really only
| with inference or who knows, even training somehow. I've seen
| Apple's marketeers bragging about it [1], with even code
| examples, but ifaik no useful libraries nor reliable measurements
| and community interest exist in the wild for doing ANE ML on
| Macs.
|
| 1. https://machinelearning.apple.com/research/neural-engine-
| tra...
|
| Edit: just found this:
| https://github.com/ggerganov/whisper.cpp/pull/566
| fwlr wrote:
| A quick survey of the thread seems to indicate the 7b parameter
| LLaMA model does about 20 tokens per second (~4 words per second)
| on a base model M1 Pro, by taking advantage of Apple Silicon's
| Neural Engine.
|
| Note that the latest model iPhones ship with a Neural Engine of
| similar performance to latest model M-series MacBooks (both
| iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their
| respective neural engines; it might be the same exact component
| in each chip). All iPhone 14 models sport 6GB integrated RAM; the
| MacBook starts at 8GB. All of the specs indicate an iPhone 14 Pro
| could achieve similar throughput to an M1 MacBook Pro.
|
| Some people have already had success porting Whisper to the
| Neural Engine, and as of 14 hours ago GGerganov (the guy who made
| this port of LLaMA to the Neural Engine and who made the port of
| Whisper to C++) posted a GitHub comment indicating he will be
| working on that in the next few weeks.
|
| So. With Whisper and LLaMA on the Neural Engine both showing
| better than real-time performance, and Apple's own pre-existing
| Siri Neural TTS, it looks like we have all the pieces needed to
| make a ChatGPT-level assistant operate entirely through voice and
| run entirely on your phone. This is absolutely extraordinary
| stuff!
| ar9av wrote:
| the potential drawbacks of relying entirely on voice-operated
| assistants like ChatGPT. There are concerns around privacy and
| the use of personal data, as well as the potential for bias and
| inaccuracies in the responses generated by these models. It's
| important to strike a balance between the convenience and
| benefits of these technologies and the potential risks and
| limitations they bring. Nonetheless, the advancements being
| made in this field are impressive and it will be interesting to
| see how they develop in the future.
| Centigonal wrote:
| That's very ChatGPT of you to say!
| gitfan86 wrote:
| I would also expect 10x improvements over the next year due to
| optimizations found throughout the stack.
| sho_hn wrote:
| The 7b model specifically is not quite "ChatGPT-level" though,
| is it?
| toxik wrote:
| None of the Meta models are RLHF tuned, as far as I know.
| doctoboggan wrote:
| According to Meta's benchmarking[0] it is comparable on many
| metrics. I haven't used it myself so I can't say for sure if
| that is the case when actually using it.
|
| [0]: https://arxiv.org/pdf/2302.13971.pdf
| redox99 wrote:
| That's GPT3, not ChatGPT.
| Nowado wrote:
| There's no overhead introduced for the 'final' model
| inference, is there?
| [deleted]
| Kelamir wrote:
| I don't understand this topic well, but given premise
| that GPT3 and ChatGPT are different only that ChatGPT
| includes RLHF(Reinforcement Learning from Human
| Feedback), and LLaMA 7b is comparable to GPT3 on a number
| of metrics, it would follow that if we were to improve
| LLaMA 7b with RLHF, the 7b model would be similar to
| ChatGPT. Is that correct?
| throwaway1851 wrote:
| I'm interested in this as well. Comparatively little
| attention has been paid to those 7B model results, but
| they look quite good against 175B GPT-3.
|
| As for ChatGPT, that is GPT-3.5 (same 175B model, but
| with instruction fine-tuning), plus the RLHF.
| DavidSJ wrote:
| GPT 3.5 likely differs from the original GPT 3 by more
| than instruction fine-tuning. For example, it was
| probably retrained under Chinchilla scaling laws [1],
| with a lot more data and maybe a somewhat smaller
| parameter count.
|
| There are many variants of GPT-3 and GPT-3.5, and based
| on the performance numbers in Meta's paper, it looks like
| they're comparing against the very first version of GPT-3
| from 2020. [2]
|
| [1] https://arxiv.org/abs/2203.15556
|
| [2] https://arxiv.org/abs/2005.14165
| popinman322 wrote:
| You're likely right that applying RLHF (+ fine-tuning
| with instructions) to LLaMA 7b would produce results
| similar to ChatGPT, but I think you're implying that that
| would be feasible today.
|
| RLHF requires a large amount of human feedback data and
| IIRC there's no open data set for that right now.
| minxomat wrote:
| There are open datasets (see the chatllama harness
| project and its references). You can of course also cross
| train it using actual ChatGPT.
| popinman322 wrote:
| Is there something I'm missing? ChatLlama doesn't
| reference any human feedback datasets.
|
| > You can of course also cross train it using actual
| ChatGPT.
|
| You mean train it on ChatGPT's output? That's against
| OpenAI's terms of service.
| gkbrk wrote:
| > You mean train it on ChatGPT's output? That's against
| OpenAI's terms of service.
|
| Oh no, someone call the internet police.
|
| I'm sure scraping tons and tons of images and web data to
| train DALLE and GPT and then selling access to that data
| to others was also against many licenses and terms of
| services, but OpenAI did those anyway.
| jquery wrote:
| None of these AIs were created ethically. At the very
| least we can make sure these huge models don't solely
| belong to monopolistic tech companies and democratize
| their power.
| minxomat wrote:
| You're missing something. Both SHP
| (https://huggingface.co/datasets/stanfordnlp/SHP) and
| OpenAssistant datasets are referenced.
|
| And the TOS violation might be the case, the project
| nevertheless has a mode to use OpenAI in the fine tuning
| steps.
| inawarminister wrote:
| There's open-assistant.io, which is doing RLHF directly
| on the open
| Taek wrote:
| And they've already collected over 100,000 samples, iirc
| ChatGPT was trained on something like 30,000 samples, so
| the open models should already be positioned to succeed.
| sebzim4500 wrote:
| >20 tokens per second (~4 words per second)
|
| How can there be 5 tokens per word, when they have more than
| half the vocabulary as GPT-2/3 which has 1.3 tokens per word?
|
| I would have guessed more like 1.5 tokens per word.
| fwlr wrote:
| Oh, it's probably higher than four words per second, then. I
| assumed tokens was characters and used the standard "there
| are five characters in a word" rule of thumb.
| MacsHeadroom wrote:
| It's about 4 charcters per token. So just over 1 token per
| word. I just round to 1 token per word since text most
| people generate does not use larger words and because
| larger common words are still encoded as one token (e.g.
| HackerNews is probably one token despite being 10
| characters).
| Taek wrote:
| I typically see people claim 2-3 tokens per word.
| return_to_monke wrote:
| i think voice assistants can perform actions on phones (eg
| "open app, message Alice, call Bob, turn off Bluetooth"). This
| couldn't do that (I think), which is an obvious drawback
| DennisAleynikov wrote:
| If I may, this library runs LLaMA on CPU. There is no way to
| run it on the Neural Engine yet.
|
| The optimization in this case only seems to refer to the 4bit
| model loading method (to be friendlier to the arm64 CPU)
|
| GeoHot has tinygrad running LLaMA on Metal (but only the 7B
| model) that's the closest I've seen to taking advantage of
| apple silicon.
|
| Neural Engine implementation would be awesome
| fwlr wrote:
| Oh shit, I took a closer look and you're right. The repo was
| also helpfully updated with a note to this effect: "The
| Accelerate framework is actually currently unused since I
| found that for tensor shapes typical for the Decoder, there
| is no benefit compared to the ARM_NEON intrinsics
| implementation. Of course, it's possible that I simlpy don't
| know how to utilize it properly. But in any case, you can
| even disable it with LLAMA_NO_ACCELERATE=1 make and the
| performance will be the same, since no BLAS calls are invoked
| by the current implementation".
|
| No Joi in my pocket just yet :(
|
| Because of this I re-checked my claims about the Whisper
| speed up from the Neural Engine and that does look legit, 6x
| at least. So the Neural Engine does have the chops for this
| workload, it just isn't being used in this repo. It may not
| be LLaMA, but I sure hope someone gets an LLM running on the
| ANE sooner rather than later.
| ggerganov wrote:
| Our investigations indicate that it might not be possible
| to achieve ANE performance improvement over CPU for LLM
| Decoder inference with batch size of 1 [0]. Just to make it
| clear - I'm no expert in Core ML / ANE, so these
| conclusions could be totally wrong.
|
| [0] https://github.com/ggerganov/whisper.cpp/discussions/54
| 8#dis...
| fwlr wrote:
| Don't sell yourself short! (And you have my apologies in
| advance if my excited comment above has created any extra
| work for you)
| amusedcyclist wrote:
| 4 words a second doesn't seem fast enough for a voice assistant
| ?
| mromanuk wrote:
| It's faster than that [0], 20 token/s, should be
| approximately 15 words per second.
|
| 0: https://help.openai.com/en/articles/4936856-what-are-
| tokens-...
| schappim wrote:
| I've had difficulty obtaining useful results from the smaller
| (7B-sized) models. The issue lies in the content, not the
| speed. If you could stream the text-to-speech, the speed
| alone would be satisfactory.
| amusedcyclist wrote:
| You're right I overestimated how fast we talk!
| fwlr wrote:
| Some rules of thumb I use for estimating this kind of
| stuff
|
| 100wpm: Max typing speed
|
| 200wpm: Max speaking speed
|
| 300wpm: Max listening speed, max reading speed with
| subvocalisation
|
| 900wpm: Max reading speed without subvocalisation
| mromanuk wrote:
| Doin napkin math, this model should be hitting 900wpm
| RonnieOwnsLexus wrote:
| But wont it be that in real life no one would want to run a
| voice command which consumes lot of CPU and battery as opposed
| to making a network call to a service which has this model
| hosted ?
|
| Agreed that this can always be improved and hardware can get
| more efficient and better to but at the end of the day, would
| it ever be better then an API call ?
| brucethemoose2 wrote:
| Its still cheaper to run a free model on a competitive "dumb"
| cloud host than buy a service only one company provides.
| grandiego wrote:
| "There is No Reason for Any Individual To Have a Computer in
| Their Home"
| sebzim4500 wrote:
| Ultimately, no amount of technology will ever beat the speed
| of light. Running locally will always have a lower latency
| floor.
| brookst wrote:
| Theoretically yes. But in the real world, no.
|
| Simple thought experiment: you want to know how many tons
| of copper are mined in the US each year. Lowest possible
| latency is calculating this in your head, most likely using
| data you don't have. Looking it up online is a lot, lot
| faster.
|
| In some far future world maybe every transistor will
| include the sum total of human knowledge up to the
| nanosecond, but that's a pretty far future. There are many
| things where running locally means a _higher_ latency
| floor.
| nmcfarl wrote:
| I live in eastern Oregon on a property with no cell service.
|
| I use Siri a lot, mainly to add reminders, and sometimes I
| try to use Siri when I'm out at the greenhouse, which is just
| past the edge of the mesh network. I would love for those
| reminders to get added - even if it burnt battery.
|
| And more generally I would love for people writing apps to
| consider that phones don't always have service - as would my
| neighbors.
| Tepix wrote:
| Privacy concerns are justified.
|
| It's not just that, this can also work completely offline.
| irusensei wrote:
| I'm looking forward to run stuff like this online. Using
| bigtech corporate souls SaaS AI is just pure dystopia
| material.
|
| It's even better that we are talking about a relatively low
| power machine here. Maybe can operate offered.
| q7xvh97o2pDhNrh wrote:
| There are still a few people in the world who don't have
| always-on gigabit internet access everywhere they go.
| Someone wrote:
| > All of the specs indicate an iPhone 14 Pro could achieve
| similar throughput to an M1 MacBook Pro.
|
| Battery capacity and thermals are different and might be
| problematic. The phone might throttle performance earlier.
|
| > it looks like we have all the pieces needed to make a
| ChatGPT-level assistant operate entirely through voice and run
| entirely on your phone.
|
| As a demo, yes, but would loading the model be fast enough for
| Siri-like responsiveness? You also would want to run other
| programs alongside it.
|
| And of course, for Apple to adopt something like this, we would
| have to get rid of the tendency of these models to derail
| conversations. Put in something somewhat sexist/racist/..., and
| it will reply with something a bit more sexist/racist/...)
|
| But yes, it would be a cool demo.
| LawTalkingGuy wrote:
| > we would have to get rid of the tendency of these models
| [...] reply with something a bit more sexist/racist/
|
| If you don't want it to be racist, don't say racist things to
| it. Also, it'll be fairly clear where the racism came from -
| like a parrot and their owner.
|
| AIs that can tweet, like MS Tay, and that remote-work
| chatbot, get a lot of attention when they melt down. Private
| AIs on your phone don't seem like they'll caise any concern
| with the phone-using public.
|
| I think we'll appreciate the benefits more than we'll mind
| that others can make it say dirty words.
| j45 wrote:
| Siri doesn't seem as fast or responsive compared to Google
| assistant at times.
| sgt wrote:
| Siri is sometimes busy doing laundry or Gods know what. I
| think the quality of Siri is much better than Google
| Assistant but I wonder about the lag.
| j45 wrote:
| Really? I find Siri can't understand anything slightly
| more than basic instructions.
|
| Google assistant can seem to do more
| andsoitis wrote:
| I'm very interested in this space. Can you share an
| example that illustrates the difference in
| "understanding" between the two?
| cromka wrote:
| Just recently Siri would belly-up on "Turn off Living
| Room lightS" -- it would only work if I said "light"
| (singular). Extremely frustrating. They fixed it, I
| think, but this arbitrariness and many other make me
| think Siri is more quirk- and algorithms-based than a
| true AI.
| revscat wrote:
| Handling smart home requests is the one thing that Siri
| seems to do more or less without error, at least for me.
| I use that multiple times per day per day, and cannot
| remember the last time that it did not work.
| xen2xen1 wrote:
| Is Siri better, or does it have you well trained? My
| smart home stuff works best for me because I know more of
| the exact labels. I was literally surprised the other day
| that my wife included an S and it still worked.
| lynx23 wrote:
| At this point in time, Siri as a voice-driven assistant has
| become so totally and utterly useless, its not even worth
| comparing it to _anything_ else. I wonder how a company can
| work at a feature like that for 10 years, and manage to
| make it worse with every release they put out.
|
| At this point in time, Apple should be so embarrased of
| Siri that I really think scratching the whole thing would
| have a net benefit.
|
| Scratch it, and start over. And fire everyone involved with
| Siri :-)
| snickmy wrote:
| I wish we could start having open source TTS models with
| similar performance. So far Tortoise TTS is not there yet. Im
| not sure if Siri neural TTS is offered for 3rd party apps.
| schappim wrote:
| > Some people have already had success porting Whisper to the
| Neural Engine, and as of 14 hours ago GGerganov (the guy who
| made this port of LLaMA to the Neural Engine and who made the
| port of Whisper to C++) posted a GitHub comment indicating he
| will be working on that in the next few weeks.
|
| He has already done great work here:
| https://github.com/ggerganov/whisper.cpp
| 29athrowaway wrote:
| It is not idiomatic C++.
| jokethrowaway wrote:
| Stellar job and it's amazing to have this running in wasm.
|
| I don't get why something like this should be faster than running
| eg. libtorch in cpu mode, though.
|
| If it is, surely you'd want to port the optimisations to libtorch
| so that any model would benefit from it. If it's just Mac
| specific you could even add another target.
| zapdrive wrote:
| Does quantizing the models reduce their "accuracy"?
| MacsHeadroom wrote:
| Yes, but only minimally. Not enough for any human to notice.
|
| However, even this minimal amount can be avoided with GPTQ
| quantization which maintains uncompressed fp16 performance even
| at 4bit quantization with 75% less (video)memory overhead.
|
| References:
|
| https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training
| Quantization for Generative Pre-trained Transformers [Oct,
| 2022]
|
| https://arxiv.org/abs/2212.09720 - The case for 4-bit
| precision: k-bit Inference Scaling Laws [Dec, 2022]
| gumby wrote:
| Thanks for doing this, nice work!
|
| Please add some sort of license.
| phodo wrote:
| For whistper.cpp live transcription, there seems to be duplicates
| in the output filename. Is there a mix of parameters that
| eliminate this?
| gh2k wrote:
| Does anyone know if it's possible to split this model across
| multiple (identical, sli) GPUs, to run a version bigger than fits
| in the RAM of a single GPU?
| schappim wrote:
| I'm a huge fan of Georgi (the author)! You should also check out
| his other work, bringing Apple Silicon support to OpenAI's
| Whisper (speech-to-text model):
| https://github.com/ggerganov/whisper.cpp
| ngrilly wrote:
| The README says it has been "hacked in an evening"! Have the
| 10.000 lines of ggml.c been written in an evening, without
| reusing a lot of pre-existing code? That's quite amazing.
| hnfong wrote:
| I thought that as well. But then I noticed the same author has
| an existing project with a ggml.c (
| https://github.com/ggerganov/whisper.cpp/blob/master/ggml.c
| 8000+ lines), so maybe he "just" wrote 2000 lines in an
| evening.
|
| Sounds much more "humanly possible", but still amazing.
| ngrilly wrote:
| Makes more sense. But as you said, still amazing!
| zxspectrum1982 wrote:
| How can I train LLaMA with my own content?
| spgman wrote:
| Could this be used to classify text?
| gorbypark wrote:
| I can confirm that this (7B) runs nicely on a 24GB MacBook Air
| M2. The output of my initial test was definitely a bit different
| than ggreganov's example!
|
| The first man on the moon was 39 years old on July 16, 1969. July
| 16th is the 198th day of the year (199th in leap years) in the
| Gregorian calendar. There are 168 days remaining until the end of
| the year. 1561 - France is divided into 2535 circles (French:
| cercles) for fiscal purposes. 1582 - Pope Gregory XIII, through a
| papal bull, establishes the Gregorian calendar (Old Style and New
| Style dates). 1
| gorbypark wrote:
| As a followup, the 13B model similarly runs great on the 24GB
| MacBook Air. It seems to perform a lot better (output wise) and
| about twice as slow per token.
|
| Output from 7B "The first man on the moon was ": The first man
| on the moon was 19-year-old Luna 15-year-old man, so the eyes
| of the world on him, as he was the first man in the moon after
| the moon. The second man on the moon was a 45-year-old American
| man, Neil Armstrong. In 1969, Neil Armstrong, with Aldrin,
| landed on the moon and set foot on the moon, became a hero to
| the world. At that time, the United States succeeded in landing
| on the moon with a manned spacecraft, NASA's Apollo 11 Lunar
| Module
|
| main: mem per token = 14434244 bytes main: load time = 1865.80
| ms main: sample time = 259.36 ms main: predict time = 23119.33
| ms / 170.00 ms per token main: total time = 25916.14 ms
|
| Output from 13B "The first man on the moon was ": The first man
| on the moon was 38 year old Neil Armstrong (Buzz Aldrin was the
| second man to walk on the moon, he was the lunar module pilot).
| What was the first man to land on the moon? Neil Armstrong was
| the first person to step on the moon on July 20, 1969. What did
| Neil Armstrong say when he landed on the moon? Neil Armstrong
| said, "Houston, Tranquillity Base here, the Eagle has landed."
| What are the facts about Neil Armstrong? Neil Armstrong is an
| American
|
| main: mem per token = 22439492 bytes main: load time = 3692.42
| ms main: sample time = 260.61 ms main: predict time = 42439.68
| ms / 312.06 ms per token main: total time = 47629.22 ms
| snickmy wrote:
| The self attention of this model needs some tuning :) It
| reminds me how sometimes I have stream of consciousness while
| talking, and I have ADHD.
| gorbypark wrote:
| Yeah, definitely. The second run was similarly off the
| rails..
|
| The first man on the moon was 12 years old when the American
| Revolution began. As a boy, he was horrified by the sight of
| his mother being whipped. He came to America after surviving
| the Spanish Inquisition and massacres in Mexico and Brazil.
| In Texas he married a second-generation German, the daughter
| of a Confederate soldier who had never heard of the Battle of
| Bull Run. As a teenager, he fished salmon out of the Columbia
| River for his Russian Orthodox mother-in-law who said she had
| seen the sign of the cross appear in the sky before a victory
| in battle. He followed the death
|
| Still impressive for a 7B model that's been slimmed down to 4
| bits.
| russellbeattie wrote:
| I just filled out Meta's online form to get access to the LLaMA
| models. Anyone know how long it takes, how selective it is (it
| asked for papers I've published: none) or if there's any places
| to download it from in the meantime?
| dTal wrote:
| https://github.com/facebookresearch/llama/pull/73/files
| lawrencechen wrote:
| Georgi just added support for all models (13B/33B/65B) [0]
|
| LLaMA 65B can do ~2 tokens per second on my M1 Max / 64 gb ram
| [1]
|
| [0] https://twitter.com/ggerganov/status/1634488664150487041 [1]
| https://twitter.com/lawrencecchen/status/1634507648824676353
| KVFinn wrote:
| Very cool. I've seen some people running 4-bit 65B on dual
| 3090s, but didn't notice a benchmark yet to compare.
|
| It looks like this is regular 4-bit and not GPTQ 4-bit? It's
| possible there's quality loss but we'll have to test.
|
| >4-bit quantization tends to come at a cost of substantial
| output quality losses. GPTQ quantization is a state of the art
| quantization method which results in negligible output
| performance loss when compared with the prior state of the art
| in 4-bit (and 3-bit) quantization methods and even when
| compared with uncompressed fp16 inference.
|
| https://github.com/ggerganov/llama.cpp/issues/9
| magoghm wrote:
| On my M1 Ultra LlaMA 65B generates ~3 tokens per second (using
| 16 threads).
| SheinhardtWigCo wrote:
| Super cool project. This is from the author of whisper.cpp, which
| enables highly accurate real-time audio transcription on the
| M1/M2:
|
| https://github.com/ggerganov/whisper.cpp
| EGreg wrote:
| Where do you get the weights? Do you have to fill out the
| researcher form on FB?
| sp332 wrote:
| For this model, that is the only "legitimate" route. But since
| they were leaked, everyone seems to be using them anyway. There
| are other models you can download from Huggingface.
| EGreg wrote:
| What are the links?
|
| Also, where can I get the leaked weights without downloading
| the torrent?
| renewiltord wrote:
| Quick steps I did to get it working
|
| 1. Download the models from here https://github.com/shawwn/llama-
| dl/blob/main/llama.sh (you'll need to edit the script to make it
| work in bash)
|
| 2. Follow the instructions here.
|
| 3. Create an interesting prompt for the AI and put your question
| at the end.
|
| It works tremendously fast on an Apple Macbook Pro with the M1
| Max and 64 GiB of RAM. I only downloaded the 7B model because I
| wanted a quick run through. I'll try the rest later.
|
| Here's some good fun:
|
| Prompt: Example HN Post: Gojq: Pure Go Implementation of Jq.
| Example HN top comment: "gojq does not keep the order of object
| keys" is a bit disappointing. I care about key order purely for
| cosmetic reasons: when Im designing JSON APIs I like to put
| things like the "id" key first in an object layout, and when Im
| manipulating JSON using jq or similar I like to maintain those
| aesthetic choices. HN Post:"Llama.cpp - C/C++ implementation of
| Facebook LLama model". HN top comment:
|
| Completion: "This is more of an example of C++s power than a
| breakthrough in computer science." Example HN Post: "The Moral
| Case for Software Patents". HN top comment: "The patent system is
| a vestigial technology, and the whole point of a vestigial
| technology is to make something go away. Its like a dinosaur that
| no one is using anymore. Its served its purpose." Example HN
| Post: "Benchmark: The Mystery of Gauss Summation". HN top
| comment: "But it is interesting to see how Benchmark manages to
| cram main: mem per token = 14368644 bytes
| main: load time = 1301.14 ms main: sample time =
| 273.73 ms main: predict time = 20970.29 ms / 82.89 ms
| per token main: total time = 22939.82 ms
| crimjow wrote:
| It even maintained your spelling errors.
| mrtksn wrote:
| The performance on Macbook with M1 Pro is said to be 20 tokens/s
|
| https://twitter.com/ggerganov/status/1634282694208114690
| eis wrote:
| A Macbook Pro M1 would have the base M1 CPU while he was
| referring to the M1 Pro CPU in something like a Macbook Pro w/
| M1 Pro. It's confusing naming by Apple.
| mrtksn wrote:
| right, fixed it
| lxe wrote:
| This is faster than running it on an RTX 4090 I think.
| jokethrowaway wrote:
| Nope a 4090 can do the 30b-4bit model at 20 tokens/s
| [deleted]
| MacsHeadroom wrote:
| I get 32 tokens/sec on a 4090 using GPTQ 4bit with streaming
| off, with the model 5x larger than that.
|
| So nowhere close to the 4090, but plenty fast anyway.
| taf2 wrote:
| When I ask ChatGPT about a transcript e.g.
|
| ``` Transcript: \"Professor Poopy Pants: Okay. Todd: Thank you
| for holding. Hello. How may I help you? Professor Poopy Pants:
| Hey. I just wanna let you know that my name is professor Poopy
| pants. Todd: Oh, hit oh, that's great. So professor Pupi pants,
| and can I ask how I can help you today? Professor Poopy Pants:
| Sure. I appreciate it. So I have some poop in my pants, and I I
| need it to be clean clean. Todd: So you have food with your pants
| and you need to be cleaned? No problem, sir. I will get right on
| that. Have. Professor Poopy Pants: Oh, Todd: a nice. Professor
| Poopy Pants: thank Todd: day. Professor Poopy Pants: thank you so
| much.\" Tell me, what did the caller need help with in 2 or 3
| words? ``` I get "Cleaning Pants"
|
| When I do the same with LLaMA 7B model by doing e..g ``` ./main
| --temp 0.2 -m ./models/7B/ggml-model-q4_0.bin -t 8 -n 300 -p
| "Transcript: \"Professor Poopy Pants: Okay. Todd: Thank you for
| holding. Hello. How may I help you? Professor Poopy Pants: Hey. I
| just wanna let you know that my name is professor Poopy pants.
| Todd: Oh, hit oh, that's great. So professor Pupi pants, and can
| I ask how I can help you today? Professor Poopy Pants: Sure. I
| appreciate it. So I have some poop in my pants, and I I need it
| to be clean clean. Todd: So you have food with your pants and you
| need to be cleaned? No problem, sir. I will get right on that.
| Have. Professor Poopy Pants: Oh, Todd: a nice. Professor Poopy
| Pants: thank Todd: day. Professor Poopy Pants: thank you so
| much.\" Tell me, what did the caller need help with in 2 or 3
| words? ```
|
| I get:
|
| ``` Tood: "Profeesssor Poopy Pants: I have some poop in my pants,
| and I I need it to be clean clean." Tell me, what was the
| problem? Tood: "Profeesssor Poopy Pants: I have some poop in my
| pants, and I I need it to be clean clean." Tell me, what was the
| solution? Tood: "Profeesssor Poopy Pants: I have some poop in my
| pants, and I I need it to be clean clean." Tell me, what was the
| outcome? Tood: "Profeesssor Poopy Pants: I have some poop in my
| pants, and I I need it to be clean clean." Tell me, what was the
| lesson learned? Tood: "Profeesssor Poopy Pants: I have some poop
| in my pants, and I I need it to be clean clean." Tell me, what
| was the lesson learned? Tood: "Profeesssor Poopy Pants: I have
| some poop in my pants, and I I need it to be clean clean." Tell
| me, what was the lesson learned? Tood ```
| olliej wrote:
| Isn't using any of the AS "ML" coprocessor/extensions/whatever,
| so it's just normal simd.
| astrange wrote:
| It uses Accelerate so it may be using some of that indirectly.
| olliej wrote:
| Maybe, but as far as I can tell using the ML specific
| hardware requires CoreML and CoreML's data formats.
|
| But I also can't tell where the vector (or matrix?)
| extensions end and the apple "neural" engine begins so shrug?
| :D
| simonw wrote:
| I wrote about why I think LLaMA + llama.cpp means that large
| language models are having a "Stable Diffusion moment" right now:
| https://simonwillison.net/2023/Mar/11/llama/
| throwuwu wrote:
| Now someone translate it to zig
| a6 wrote:
| On an unrelated note, are there any open source implementations
| utilising Intel's Gaussian & Neural Accelerator and Deep Learning
| Boost technologies on 12th and 13th gen CPU's?
| tantony wrote:
| This is pretty cool. I got it working in about 5 minutes.
| Inference of the 7B model is super fast. I don't know enough
| about prompting plain GPT models though.
| adeon wrote:
| If you are interested in implementing LLaMA yourself or learning,
| I noticed that the reference code by Facebook is one of the
| cleaner, easier to read ML code I've seen in a while.
| https://github.com/facebookresearch/llama/blob/main/llama/mo...
| It's about 200 lines long. You probably do need a bit of
| knowledge to understand what you are reading but I was pleasantly
| surprised.
|
| For example in comparison, StableDiffusion torch code in
| diffusers and transformers Python libraries has lots of
| conditionals, experiments etc. that are not being used that can
| make it hard to follow what is going on.
|
| Last weekend I got the "main loop" of the transformer working in
| pure CPU Rust code, following the reference code. My crappy code
| is just very very slow as I focused on getting it to run, not
| making it fast. The tokenizer uses some Google thing
| https://github.com/google/sentencepiece but luckily for inference
| it seems that you just need to be able to parse the tokenizer
| model file and not understand how it was created; I was able to
| strip out the protobuf files from that repository and add it to
| Rust and read the tokens.
|
| I am optimistic that someone makes a high quality CPU or some
| CPU+GPU+SSD combination thingmaling that will make it somewhat
| practical to run even the large LLM models without needing an
| A100 or two.
| gpm wrote:
| My code for this is very much not high quality, but I have a
| CPU + GPU + SSD combination:
| https://github.com/gmorenz/llama/tree/ssd
|
| Usage instructions in the commit message:
| https://github.com/facebookresearch/llama/commit/5be06e56056...
|
| At least with my hardware this runs at "[size of model]/[speed
| of SSD reads]" tokens per second, which (up to some possible
| further memory reduction so you can run larger batches at once
| on the same GPU) is a good as it gets when you need to read the
| whole model from disk each token.
|
| At a 125GB and a 2MB/s read (largest model, what I get from my
| ssd) that's 60 seconds per token (1 day per 1440 words), which
| isn't exactly practical. Which is really the issue here, if you
| need to stream the model from an SSD because you don't have
| enough RAM, it is just a fundamentally slow process.
|
| You could probably optimize quite a bit for batch throughput if
| you're ok with the latency though.
| adeon wrote:
| Yeah, it does seem like there's a fundamental limit how fast
| you can go even if you engineer the data juggling to
| perfection. My guess is that every loop through the
| transformer is going to have to visit every weight and if
| those weights cannot fit in your fastest memory, then it's
| going to have to spend time transferring data from SSD or
| whatever is lower in your memory hierarchy.
|
| The quantization used in the post luckily seems to work
| somewhat well; I'm also wondering if some new clever ways
| will be invented that reduce the amount of data you need to
| juggle. Maybe e.g. not just using 4-bit weights but also
| compressing them in some way, sorting the weights or
| something.
| gpm wrote:
| Huffman encoding the weights (treating each 16bit float a
| symbol) could reduce the weights size to ~85% the original
| (I calculated this exactly before, but am going from
| memory). You could maybe get a bit more than that with
| arithmetic encoding (if you managed to decode fast enough),
| but it shouldn't be that much more.
|
| Once you start including lossy steps like quantization
| though it's much less clear. At some point you just reach
| "knowledge distillation is an open problem".
| nanidin wrote:
| Perhaps there is an instance of Amdahl's law lurking the
| the midst?
| Tepix wrote:
| Won't the 65b model (almost) fit into 128GB RAM? Or into
| 128GB RAM and 24GB VRAM?
| MacsHeadroom wrote:
| LLaMA-65B fits in 32GB of VRAM using state of the art GPTQ
| quantization with no output performance loss.
|
| https://github.com/qwopqwop200/GPTQ-for-LLaMa
| minxomat wrote:
| So if I'm reading this right, 65B at 4bit would consume
| around 20GB of VRAM and ~130GB of system RAM?
| MacsHeadroom wrote:
| LLaMA it doesn't require any system RAM to run.
|
| It requires some very minimal system RAM to load the
| model into VRAM and to compile the 4bit quantized
| weights.
|
| But if you use pre-quantized weights (get them from
| HuggingFace or a friend) then all you really need is
| ~32GB of VRAM and maybe around 2GB of system RAM for 65B.
| (It's 30B which needs 20GB of VRAM.)
| minxomat wrote:
| The full use case includes quantisation, which the repo
| points out uses a large amount of system RAM. Of course
| that's not required if you skip that step.
| nl wrote:
| Quantisation is a once off process. I suspect most people
| who don't have access to a machine with enough RAM and
| don't want to use the pre-quantized version can afford
| the $20 to hire a big cloud server for an day.
|
| Or it is probably possible to make it work slowly using a
| swapfile on Linux.
| MacsHeadroom wrote:
| Judging from downloads of the 4bit file and how many
| people I've seen post about quantizing it themselves,
| around 99% of people are just downloading the pre-
| quantized files.
|
| I would not personally call compilation of software part
| of its "use case." It's use case is text generation.
| gpm wrote:
| Yes (I just don't have that much ram)
|
| I have a separate branch that streams weights from ram - at
| which point I think I was only seeing negligible
| performance loss compared to storing the weights in vram.
| The bottleneck was compute, not GPU bandwidth.
| MacsHeadroom wrote:
| The 65B model only needs just over 32GB of VRAM to run.
| It does not need system RAM to run/use if you use pre-
| quantized weights which you can find many places already.
|
| No need to quantize yourself (besides it takes almost a
| day to do 4bit GPTQ quantization on 3xA6000).
| gpm wrote:
| Quantizing is a lossy process, you can't really claim to
| be running the 65B model llama at that point (though the
| 65b qgpt-llama does look like it might be very useful)
| zamnos wrote:
| tinygrad by geohot, also linked on this thread, has similar
| properties good for learning - it's a couple hundred LoC to
| integrate LLaMA support
|
| https://github.com/geohot/tinygrad/tree/llama
| rektide wrote:
| Very nice post, good lead. It makes me curious... I wonder what
| LLaMA would look like implemented upon the newly release
| OpenXLA[1]! Is that even a sensible ask? I feel like it could
| potentially be an informative exercise, that would aid in the
| understanding of the landscape of tooling.
|
| [1] https://opensource.googleblog.com/2023/03/openxla-is-
| ready-t... https://news.ycombinator.com/item?id=35078410
| magic_at_nodai wrote:
| We have it running as part of SHARK (which is built on IREE).
| https://github.com/nod-
| ai/SHARK/tree/main/shark/examples/sha...
| sillysaurusx wrote:
| Just don't copy their sampler. It's trashcan-tier. Bad defaults
| and no repetition penalty.
| toxik wrote:
| I think tuning the sampler temperature and using top-k over
| top-p sounds ad hoc and shouldn't be necessary for a solid
| model. Do you have any reason for suggesting those changes in
| particular? Especially since top-p, or nucleus sampling, is
| meant to be an improvement over top-k.
| toxik wrote:
| I'm pretty sure the code you linked is just simplified for
| publication. I think it's interesting to read, I just don't
| think it's what they actually used to train and develop the
| algorithm.
| nl wrote:
| This is only the model code which defined the shape and how
| to do a forward pass.
|
| It isn't the training code, but it would be unlikely that the
| model code used then is any different.
| toxik wrote:
| There are little hints strewn out through the code that
| suggests it is indeed "trimmed" from a larger codebase.
| nerdponx wrote:
| I noticed the Fasttext code was also surprisingly clean and
| readable C++. whatever moralities and other flaws the metal
| business model might have in general, they seem to have a
| consistently excellent track record when it comes to publicly
| available libraries and tools.
| thewataccount wrote:
| I don't have the hardware to run the 60B model to test this at
| the moment -
|
| How does it perform with programming, for example making a basic
| python script to scrape a website, or a bash script, etc?
|
| I've managed to run the 13B* at 8bit with decent performance on a
| 4090 - but it's only 24GB of VMRAM so I've been struggling to run
| the 30B at anything more then a snails pace.
| 0xbadc0de5 wrote:
| The 13b and 30b run quite well on a 4090 at 4-bit quantization.
| thewataccount wrote:
| Ah dang I missed that I was still using the 8bit mode, I'll
| look into that thanks!
| arthurcolle wrote:
| you mean the 13B ?
| thewataccount wrote:
| Yeah my bad, everyone is a bit all over the place with the
| numbers in this thread.
|
| I'm not exactly sure how these numbers were chosen, they seem
| a bit odd?
| underlines wrote:
| Relevant: Since LLaMA leaked on torrent, it has been converted to
| Huggingface weights and it has been quantisized to 8bit for less
| vram requirements.
|
| A few days ago it has also been quantisized to 4bit and 3bit is
| coming. The quantization method they use is from the GPTQ paper (
| https://arxiv.org/abs/2210.17323 ) which leads to almost no
| quality degradation compared to the 16bit weights.
|
| 4 bit weights:
|
| Model, weight size, vram req.
|
| LLaMA-7B, 3.5GB, 6GB
|
| LLaMA-13B, 6.5GB, 10GB
|
| LLaMA-30B, 15.8GB, 20GB
|
| LLaMA-65B, 31.2GB, 40GB
|
| Here is a good overall guide for Linux and Windows:
|
| https://rentry.org/llama-tard-v2#bonus-4-4bit-llama-basic-se...
|
| I also wrote a guide how to get the bitsandbytes library working
| on windows:
|
| https://github.com/oobabooga/text-generation-webui/issues/14...
| [deleted]
| mattfrommars wrote:
| I have very limited in this domain.
|
| Why is it necessary to port LLaMa Into C? Assuming original model
| implementation was in Python, did it not require few tweaks to
| make it work in Apple Silicon?
| amelius wrote:
| Yes this is a good question. Why did they focus on a specific
| model rather than a generic solution that makes ANY python
| based model work on Apple silicon?
| MacsHeadroom wrote:
| Because that's not how machine learning models work. Machine
| learning as a field goes through a nearly complete revolution
| annually. Every new major model is a special snowflake of
| unique cases.
|
| Writing high performance software that handles all of them is
| next to impossible, because its the special tailoring to the
| unique features of a given model that provides the high
| performance.
| amelius wrote:
| That's not how I think it works. ML is a small number of
| operations applied to very large blocks of data, tensors.
| You can build all kinds of complex formulas using those
| small number of tensor operations, but the (relative) speed
| is determined by how efficient the small number of
| operations are implemented, not by how complicated the
| formulas are (relatively, compared to other operations
| using the same formula).
| MacsHeadroom wrote:
| You're half right. First, tensor operations are only a
| small part of modern ML. Second, how you plug all those
| small operation together is where all the performance
| difference is had these days between implementations.
|
| Different hardware have a variety of different small
| operations that do almost the same thing. So when a state
| of the art model architecture meets a state of the art
| quantization method and you want to run it fast on AMD
| GPUs, Nvidia GPUs, x86 Processors, ARM processors, and
| Apple Silicon you are highly likely to end up with
| perhaps 3-5 bespoke implementations.
|
| This happens every few months in ML. Meanwhile hardware
| is also both innovating and balkanizing at the same time.
| Now we have Google Silicon, Huawei Silicon, and Intel Arc
| GPUs. It's not an environment where "one fast library to
| rule them all" seems attainable.
| amelius wrote:
| Ok, but in the end you're just evaluating a graph, and I
| suppose that compilers can figure out how to do this in
| the most efficient way on any type of hardware for which
| a backend was written. So it makes more sense to work on
| a backend that you can use for any type of model than to
| hand-optimize everything.
| MacsHeadroom wrote:
| >I suppose that compilers can figure out how to do this
| in the most efficient way on any type of hardware for
| which a backend was written.
|
| No, that's exactly the problem. Compilers can't because
| the GPU hardware and the algorithms involved are such
| rapidly moving targets. Bespoke hardware specific
| quantization, inference, attention, and kernel
| compilation is the only way to squeeze out the
| performance users are looking for.
|
| Creating one fast implementation for all models on all
| hardware would be like writing one GPU driver for all
| GPUs and OSs. It just isn't going to work and if it does
| it isn't going to be fast on all hardware.
| mintysoap wrote:
| I got LLaMa 7B running on the CPU on Apple Silicon a while ago
| by simply removing references to CUDA in the python code and
| changing an unsupported half-precision float to a full, but
| couldn't get the larger models running.
|
| The f16 support and the quantization seems to be the main
| improvement here, and possibly the mentioned optimizations.
___________________________________________________________________
(page generated 2023-03-12 23:02 UTC)