[HN Gopher] Show HN: Speeding up LLM inference 2x times (possibly)
___________________________________________________________________
Show HN: Speeding up LLM inference 2x times (possibly)
Here's a project I've been working on for the last few months.
It's a new (I think) algorithm, that allows to adjust smoothly -
and in real time - how many calculations you'd like to do during
inference of an LLM model. It seems that it's possible to do just
20-25% of weight multiplications instead of all of them, and still
get good inference results. I implemented it to run on M1/M2/M3
GPU. The mmul approximation itself can be pushed to run 2x fast
before the quality of output collapses. The inference speed is
just a bit faster than Llama.cpp's, because the rest of
implementation could be better, but with a better development I
think it can be a new method to speed up inference - in addition to
quantization. You could call it ad-hoc model distillation :) You
can change the speed / accuracy of a model at will, in real time.
Oh, and as a side effect, the data format allows to also choose how
much of the model you want to load into the memory. You can decide
to skip say 10-20-40% of the least important weights. It's
implemented for Mistral, it was also tested slightly on Mixtral and
Llama. It's for FP16 for now, but Q8 is in the works. The
algorithm is described here, and the implementation is open source.
https://kolinko.github.io/effort/ I know these are bold claims,
but I hope they survive the scrutiny :)
Author : kolinko
Score : 198 points
Date : 2024-04-17 17:26 UTC (5 hours ago)
(HTM) web link (asciinema.org)
(TXT) w3m dump (asciinema.org)
| kolinko wrote:
| Here to answer questions!
|
| A friend of mine has published the original link on HN before me,
| so I hope a double post won't be an issue :)
| dang wrote:
| We've merged them. (other one was
| https://news.ycombinator.com/item?id=40067489)
| dartos wrote:
| Thank you for this really cool and open contribution!
|
| I will be watching llama.cpp closely for them to implement this!
|
| I've been looking for ways to speed up CPU inference and I really
| like this idea of "effort"
| kolinko wrote:
| Hahah, thanks! It was a marathon to get develop this, and I'm
| glad it reached the front page.
|
| The name was proposed by chatgpt :) It claims it doesn't
| recognise this approach - so there is a chance it's really a
| new thing.
|
| I want to reach out to llama.cpp and the others - I hope it
| gets implemented. I considered just writing a patch to llama,
| but c++ and the scale of that project was beyond me.
|
| As for CPU inference - it should speed it up just as well. But
| thanks to the fact that it can load up a fraction of weights
| (e.g. just 70%, skipping the least important ones), it should
| be possible now to run models on less VRAM than before (still,
| Q8 needs to implemented though).
|
| Funnily - when I tried comparing benchmarks to llama.cpp, I
| couldn't find speeds for 7B/FP16 on MB Air 16GB, because it's
| impossible to run with regular methods. It is possible with
| Effort.
|
| Ditto, I was running full resolution, but cropped, Mixtral on
| my 96GB M2, even though it usually takes 114GB ram. I just
| loaded 75% of weights, and it was working smoothly. (before I
| messed something up with implementation and it now produces
| crap output - needs a fix)
| dhruvdh wrote:
| I would imagine the importance of weights depends on the
| prompt. How do you decide which weights are important?
| kolinko wrote:
| Yeah, that is the point more or less - it dynamically chise
| the weights layer per layer depending on the internal
| state.
|
| A bit technical explaination here.
| https://kolinko.github.io/effort/equations.html
| indymike wrote:
| > It is possible with Effort.
|
| "All things are possible with enough effort." -- Dad.
| kolinko wrote:
| Hahaha :)
| 0x4139 wrote:
| Implementing this approach could significantly enhance the
| adoption of LLMs within mobile phone libraries and other
| compact devices. I highly recommend opening an improvement
| issue for llama.cpp.
| avereveard wrote:
| we need this into llama.cpp it seems somewhat stable down to 40%
| effort
| kolinko wrote:
| Yes! I hope, if it's proven, that it will be implemented into
| the main inference engines.
|
| 40% effort is only a bit faster than a full base
| multiplication, but I hope both the speed and the accuracy
| could be improved further.
| toisanji wrote:
| this sounds related to this: https://arxiv.org/abs/2312.12456
| https://github.com/SJTU-IPADS/PowerInfer
| kolinko wrote:
| Similar theme, but they skip whole neurons, in my case it's
| down to a level of single weights.
|
| From my experiment, skipping whole neurons (so whole
| rows/columns of matrixes) didn't allow for such good results.
| In my case 30-50% neurons whole are skipped with 15% effort,
| but the rest is used partially still.
|
| There are a few papers on a similar theme that a friend sent me
| today morning - I plan to add them in the citations part
| toisanji wrote:
| awesome, looking forward to seeing how your results perform.
| I tested powerinfer on smaller models and didn't see large
| performance gains.
| bigcat12345678 wrote:
| """ So instead, let's flip the matrix, sort the elements row-
| wise, and revisit the multiplications from that direction.
|
| This is called a Compressed Sparse Row (CSR) format by the smart
| people. To do the multiplication now, we take, say, the 1 from
| the vector, multiply it by 256, and add it into the output vector
| at the 3rd row. And so on.
|
| Now, let's see what happens if we truncate the last column - the
| one with the lowest values. """
|
| How does csr works with reduced numbers multiplication?
| kolinko wrote:
| Can you rephrase the question? Not sure I get it.
| bigcat12345678 wrote:
| I could not read from the text how multiplication with CSR
| format works in the context of optimization.
|
| The key missing piece for me is that how to utilize CSR
| format to find the numbers to do multiplication, in other
| words, how does CSR format helps with picking the numbers to
| multiply with the vector.
| kolinko wrote:
| Ah, I get the question now.
|
| If I understand correctly CSR, it stores indexes and values
| as a list. I store them also as a list - that's why the
| comparison is there. The difference is that with CSR you
| store say 15% of the values from a given row. I store all
| of them, but use only the first X% of them. The X% varies
| and depends on the input vector.
|
| They are stored sorted, from the one of a highest absolute
| value to the lowest absolute value.
|
| It's after midnight so my explanations may not be too good
| now, but I hope the pseudocode on the page and the examples
| explain it slightly better.
|
| I'll be fixing grammar / typos, and asking ChatGPT to
| rewrite the page text for me tomorrow to make it more
| readable :)
| gcy wrote:
| Could you explain the pseudo code in your equations page? Is the
| second approxMul call a typo (also the capitalized V)?
|
| def precompute(W): W = W.T probes = get_probes(W) W_idx, W_val =
| sortMatrixRows(W)
|
| def approxMul(v, W_idx, W_val, probes): cutoff_chart = v * probes
| cutoff = topK(cutoff_chart, effort) approxMul(V, W_idx, W_val,
| cutoff)
| kolinko wrote:
| oh, thanks, I fixed it. No idea what I meant there originally.
|
| There are still a few typos on the page, I'll be fixing them
| tomorrow - it's midnight now, and my mental batteries are
| slowly drying out :)
| queuebert wrote:
| Please, please, please call the final product Halfwit.
|
| Seriously though, this is a very interesting biologically
| inspired idea, since not all neuronal pathways fire all the time.
|
| It seems to follow that, if you can predict which weights you
| won't need, then you should be able to compress the model
| architecture permanently, at least for certain use cases.
| kolinko wrote:
| Haha halfwit! I'm waiting for such a fork.
|
| As for predicting the weights - not necessarily so. It seems
| most weights are being used, just not all the time. Kind of
| like that saying that humans are using just 5% of their brain -
| perhaps they are, but it's various parts of the 5%.
|
| Interestingly, Effort works just as well on MoE, if not better.
| I did most of the development on Mixtral and I think it go even
| to 15-20% effort before losing quality, but there is some sort
| of a bug right now that prevents the inference on Mixtral.
|
| It's on a todo to fix, but I didn't want to delay the release
| because of it.
| HPsquared wrote:
| Halfweight. Half the weights, half the wait, half the wit.
| LorenDB wrote:
| Effortless would be another great name (since you are literally
| reducing effort to get speed). OK, maybe not "great", but "an
| option if you're going for puns".
| coolvision wrote:
| how does it compare to 8-bit/4-bit quantization in terms of
| speed/accuracy?
| kolinko wrote:
| hard to say for now, I'm curious as well, but I used simpler
| tests so far because of the implementation issues - most test
| suites are geared towards testing models and not model
| implementation.
|
| I didn't want to wait any longer with the release, but better
| tests will be coming soon I hope. Anecdotally, I think 30%
| effort should be comparable to Q8z
|
| More importantly, this algorithm should work on top of Q8. The
| quality is not yet certain though - I could use help with the
| implementation.
| kanwisher wrote:
| Could you break down a bit more about why you can skip so many
| calculations ?
| kolinko wrote:
| It's explained in detail here:
|
| https://kolinko.github.io/effort/equations.html
|
| Long story short - I kind of sort them and pick only the top %
| that would give a highest result.
|
| One part is choosing them though - I think this was done before
| in some papers. But the second part was an implementation of
| multiplication that is efficient on both gpus and cpus when
| choosing weights almost at will.
|
| All explained on the site, but I just got feedback that it may
| not be easy enough to read, so I'll push it through gpt for
| grmamar fixes soon :) It's also a bit complicated as an
| algorithm.
| throwaway2562 wrote:
| Hmmm. Very interesting. I wonder if you could speed up your
| clever approximation still further with this approach
| https://arxiv.org/pdf/2205.09120.pdf
| kolinko wrote:
| Nah, sadly this most likely will not work with Q1/Q1.5
| implementations - initially I was playing with monte carlo
| approximations (before I arrived at bucketMul), and the
| convergence was very slow for binary/ternary networks.
|
| Or, in simpler terms - if you have just ones and zeroes,
| and minus ones, you can remove zeroes from calculations,
| but that's it. No good method to figure out which ones are
| more important than the other ones.
|
| Also, there are no bits left to store positional
| information when bucketing.
|
| There are some paths that could be explored in this
| fashion, but it would require a redesign of the algorithm
| from the ground up.
| punnerud wrote:
| A bit like calculating the fastest route for a car, you
| probably don't need to calculate the distances for the opposite
| side of earth if you will not drive there. Then multiply that
| by a billion, but the optimization still holds.
| saurik wrote:
| Also being discussed at:
| https://news.ycombinator.com/item?id=40067489
| byyoung3 wrote:
| I think this is the overall idea behind the MoE LLM models right?
| MoE just expands upon this idea by learning which sets of weights
| to use
| kolinko wrote:
| I'd say that this expands on MoE really - MoE chooses
| dynamically which groups of weights may be needed, but it's
| whole matrixes. Here it's ingle weights.
|
| Also, this works on top of MoE beautifully - most of the
| development and testing was done on Mixtral and it was getting
| (anecdotally) even better results - getting down to 15-18%
| effort before seriously losing quality.
|
| I decided to release the Mistral version, but Mixtral was fully
| operational a few commits back :)
|
| Also, the cool thing - because you can load only the top say
| 70% weights, I was running Mixtral full precision on my MB 96G
| - there were no bemchmarks for this in other impls because
| others need to load full model into the memory.
|
| The real question is Q8 performance - I didn't implement it
| fully so far.
| huac wrote:
| It also feels similar to mixture of depths
| (https://arxiv.org/abs/2404.02258).
|
| Being able to apply this post-training is pretty cool though,
| makes it easier to use across a wider range of setups.
| gsuuon wrote:
| Nice writeup! Very curious about the performance per VRAM with
| this compared to just quantization. Any plans to implement a
| cross platform version?
| kolinko wrote:
| Per VRAM - not much netter, because it still uses all the
| weights, just not all the time.
|
| I mean - it can also load less weights, but quality seems to
| degrade quick after offloading more than 20-30% weights.
|
| In other words - this algorithm decouples inference time from
| VRAM use.
|
| Having said that, I'm curious as well if using effort you can
| get better results on Q8 cropped to 75% than on Q6.
|
| But it's still probably a few weeks to get the implementation
| polished enough to be well tested.
| AnthonyMouse wrote:
| > Having said that, I'm curious as well if using effort you
| can get better results on Q8 cropped to 75% than on Q6.
|
| This is what I wanted to ask. This seems like the same _kind_
| of optimization as quantization, sacrificing a bit of quality
| for performance by discarding some data. So then the question
| is, which is better, and how do they combine?
|
| You could even potentially get different results at different
| points in the scale. Maybe Q8 cropped to 75% isn't better
| than Q6 but Q4 cropped to 75% is better than Q3, or vice
| versa.
| brrrrrm wrote:
| do you have a simple python impl? :)
| kolinko wrote:
| It originally started as a fork to Recmo's cria pure numpy
| llama impl :)
|
| https://github.com/recmo/cria
|
| Took a whole night to compute a few tokens, but I used it to do
| the first tests.
|
| Also, my friend pasted the paper to claude and it produced a
| working basic impl instantly :D
|
| But in all seriousness - I think MLX implementation would be
| doable, or a wrapper to the Metal gou functionality
| kwikiel wrote:
| Will share python implementation soon as a kind of executable
| pseudo code which then can be ported to any platform.
|
| This project is kind of like ultimate nerdsnipe as math is
| quite simple, you don't need PhD to understand it and
| actually implementing things would teach you linear algebra
| faster vs just mindlessly doing exercises sets.
| kolinko wrote:
| Haha yes :) Publish it, Kacper!
|
| The project is a nerdsnipe for math geeks, because there
| are multiple small things that beg to be proven / described
| by math there. For example - what's the tradeoff between
| the number of bits we loose when embedding position vs the
| bits of information that we gain by knowing which bucket a
| weight belongs to?
|
| In other words - is it possible that when storing weights
| in the bucketed form we can actually end up having a higher
| precision than using a regular form? For Q8 we get just 4
| bits to store the weight (and 1 bit for sign, and 3 bits
| for location), but these 4 bits need to express numbers
| from a smaller range than before.
| xrd wrote:
| My takeaway is that this proves what was said on the recent
| latent.space podcast with David Luan from Adept.
|
| https://www.latent.space/p/adept
|
| "I think is going to commoditize a lot of the regular LLMs and
| soon regular multimodal models."
|
| In other words, if you train your own models, you will not get to
| take advantage of breakthroughs like this that start with open
| models (like Mistral).
|
| All the advantages are going towards the open models and this is
| an existential risk for OpenAI and other closed model companies.
| quadrature wrote:
| Maybe, but theres nothing that stops OpenAI from stealing these
| tricks.
| refulgentis wrote:
| It's extremely unlikely anyone will take any of this.
|
| Quick take:
|
| - it was 15x slower than llama.cpp when I used Apple's new
| proprietary ML framework on my MacBook
|
| - So I made it possible to skip arbitrary amounts of work.
|
| - I identified an arbitrary tradeoff that seems arbitrarily
| good to me.
|
| - I've confirmed this by making GPT-4 write some prompt with
| questions. Then I had the normal version answer, and the
| "skip arbitrary work" version answer, and it LGTM.
|
| - So I threw it up on GitHub, then on HN with a title "LLM
| inference 2x faster (possibly)", and people missed: [on my
| laptop] [in the ML framework I'm forcing myself to use]
| [based on an eval I made up] [based on an an eval where I am
| the evaluator]
|
| This *really* shouldn't have the title it does, very
| misleading.
|
| Author, please feel free to correct me, I'm sorry for not
| taking the time to find a gentler way to communicate this. I
| hope you kick ass. You did put possibly in a parenthetical,
| but its carrying the weight of the world here, people just
| see LLM 2x faster. That's why everyone is spinning off into
| grand speculation land, which I also see you valiantly
| commenting to dissuade
| kolinko wrote:
| The whole inference is slow, but it's matrix
| multiplications that count. They work reliably on all the
| Macbooks that I tested - at 50% effort it's the same speed
| as the state of the art matrix multiplications, at 25% they
| are twice as fast.
|
| The apple's MPS matrix multiplications from Apple are
| comparable in speed to the speed of Llama.cpp and the other
| models. When I was doing tests, I was comparing the
| Llama.cpp benchmarks (
| https://github.com/ggerganov/llama.cpp/discussions/4167 )
| to Apple's MPS - they match very closely. And then I was
| comparing Apple's MPS to my results.
|
| Even if the end-results would show that the models somehow
| break (which they might on Q8), there is no other
| implemented method right now that would give you such
| speedups with matrixes of 25% sparsity. The usual methods
| break even with full matrix multiplications around 15%
| mark, and show speed improvements under 10% (as far as I
| know, but I'm new to the field, so I wait to be corrected).
|
| As for the other metrics - I hope to get help from the
| community to get the implementation done properly. So far
| it's been 3 months of work 12 hours a day - even during
| Easter - to get this version going. It is as far as I can
| push it without the community support, which I'm happy I
| received over the last hour.
|
| Also, I'm not sure what you'd expect really. A full
| production ready system on the day one? From a solo
| developer? Seriously? :)
|
| Let's get the flame war going! :D
| refulgentis wrote:
| Nah it's good work, you'll be able to flame me in a
| couple weeks...months?..too when I ship my yawn-worthy
| yet another llama.cpp / OpenAI wrapper. :p
|
| I'd love this knob, particularly in llama.cpp, inference
| is a bit too slow on Android, 6 tkn/s for 3B. just can't
| stand it when people don't actually read anything but the
| title, and go crazy overboard, like, how are we in a
| thread where people are like "oh this confirm local
| models will definitely win like I heard on a podcast" and
| "big bad OpenAI will steal this".
| kolinko wrote:
| Hahah thanks, although I was hoping for a flame to get
| adrenaline flowing to push through the night :D
|
| I also hope there will be an extra knob - or more like
| knobs, because effort can be regulated smoothly layer by
| layer, token by token, matrix by matrix. Think more like
| an equalizer, not a volume control :)
|
| The biggest question right now is how (if) it will
| perform with Q8 and with smaller models. The risk is that
| the quality dropoff will show up closer to 40-60% at Q8,
| negating the performance gains.
| xrd wrote:
| I think you are commenting on the "stealing" reply and not
| my original comment. And, I think you are making my point
| stronger.
|
| OpenAI could easily (and will) put out a blog post or tweet
| saying "next models do inference 2.5x faster!" Koliko did
| that, or maybe he didn't and someone else put words in his
| mouth. I don't really care: I can validate and test your
| comments here (and they are great!) and I can try his
| experiments myself.
|
| I cannot do that against "GPT-5-2.5x faster (c) 2024"-42B
| (because it isn't released yet publicly). Putting a paper
| and some vague ideas on Arvix isn't really doing much these
| days except adding to the confusion. Truly open work like
| koliko is doing is really exciting and feels like it can
| only be done against truly open models like Mistral.
|
| Oh wait, Mistral isn't fully open either (ducks...).
| observationist wrote:
| There used to be a class of software called freeware -
| you could download and use it without restriction, you
| just couldn't resell it, or have the source to modify it.
| Llama and similar models are like freeware - an
| inscrutable binary blob crafted to work with other
| software, except instead of a VM or native OS
| environment, you have llama.cpp or similar software that
| runs the AI model.
|
| Mistral is open source, in that you can do anything the
| Apache license allows you to do, even package it into
| your own product and resell or modify it. We're missing
| the dataset details, the source to the software that
| produces the model, similar to not having access to an
| operating system and special compiler software. That's
| not a huge deal, because people don't have the resources
| to make use of those large datasets or the Mistral
| training software, which is likely highly tailored to
| their own training and development pipeline, and wouldn't
| do much good for anyone without at least a pod of A100's
| of their own.
|
| Weights available and other terms are being thrown
| around, and Meta and the like are calling their stuff
| "open" but that use of the term bears little resemblance
| to the use of the word by the open source community.
|
| The public Mistral models have open source licenses. The
| model can be used like open source software. The terms
| are permissive and free, requiring only attribution.
| Meta's license scheme is novel and not open, with
| arbitrary lawyerese and absolutely, 100% will bite
| someone in the ass when the threshold between "annoying
| to sue" and "profitable to sue" gets exceeded by someone
| using Llama in a way that's technically incorrect. Right
| now, Meta wants the goodwill more than they want a couple
| million dollars chasing a couple dozen startups.
|
| If the model doesn't have an open source license, it's
| not open. It might be freeware. Llama is freeware. You
| can, technically, do whatever you want to it, but try to
| not attract too much notice or be too successful with it.
|
| Mistral, by using Apache licensing, couldn't go after you
| even if they wanted to, unless you do something
| deliberately stupid.
| littlestymaar wrote:
| Exactly, that's the problem with the current state of things
| with open models, the players that keep their secret sauce
| keep an edge over the people doing things in open while
| benefiting from all of their work without contributing back.
| kolinko wrote:
| That was the claim with a lot of the software in the past,
| but open source won in many places in the end.
| kolinko wrote:
| In this case though, the algorithm should be just as useful to
| closed models as to open models. There is nothing there that is
| optimised specifically for Mistral - aside from the hard-coded
| dimensions in multiple places in the code :D
|
| Having said that, it's awesome to have open source models out
| there, and I hope they will ultimately win in the end.
| globally_unique wrote:
| That looks like fantastic stuff. I just want to point out the
| 15ms delay looks similar to 60Hz vsync (16.7ms), if you are
| updating the screen once per token, maybe that's causing a sync
| somehow?
| kolinko wrote:
| Nah, that's not it, I measure the CPU & GPU work separately,
| and 15ms happens between the kernel invocations. It also
| happens when I don't print out text.
|
| Thanks for the idea though! I treat it as the first community
| contribution :D
| bick_nyers wrote:
| I'm wondering if you could sort the two inputs, add the indicies
| for the multiplications together, then take the largest of that.
|
| In your 0.1 example, 1000 gets index 2, and 0.1 index 0, combines
| to 2. This will tie with the 1*8, but I think it would even out
| with larger vector lengths.
|
| Edit: I could be wrong but I think you can precompute the indices
| for the weights in advance without a prompt, then you won't need
| to perform those sorts at runtime.
| marmaduke wrote:
| Having used CSR it's not surprising, and some newer formats might
| have more mechanical sympathy like block ELL, since they avoid
| uncoalesced reads / gathers, tho the code is trickier.
| kolinko wrote:
| Oh, nice to finally bump into someone who has experience with
| CSR!
|
| bucketMul has few uncoalesced reads, and it uses a different
| data structure than the regular CSR - it's decribed here:
| https://kolinko.github.io/effort/bucketmul.html It splits each
| Matrix row into 16 parts, and chooses which ones are necessary
| to read. The writes are fully linear.
|
| Not sure if I speak sense though, it's getting a bit late
| today, and it's been a long day ;)
| uhlo wrote:
| Okay now add a small model that decides how much effort is needed
| in each inference step and we are good to go
| kolinko wrote:
| Yes! That would be awesome. Especially since there are ~32*6
| independent effort settings for every single token.
|
| I tested the most basic implementation, with a flat effort
| setting for all the muls, but I bet the results could be pushed
| even further with such an approach. Or even with just doing
| some ML to figure out which layer/matrix needs more and which
| less effort.
| uhlo wrote:
| Great work! One thing: it seems the hugging face link doesn't
| work... I get a 404
| spencerchubb wrote:
| I love this line in the gpu implementation section.
|
| "Readers fresh to GPU programming may ask now - how does it work?
|
| Readers experienced with GPU programming may ask - how the hell
| does it work?"
| kolinko wrote:
| Haha thanks! :) As far as I understand I had to implement the
| memory reads and some other things the opposite way to what is
| considered a proper approach.
|
| Would love to have that code reviewed by someone who actually
| knows stuff about Metal - this is my first gpu programming
| attempt
| smcleod wrote:
| Looks like the models are missing on Huggingface, I've logged an
| issue: https://github.com/kolinko/effort/issues/3
| kolinko wrote:
| Ah yes, forgot to make the repo public. Thanks a ton for
| pointing it out and writing the comment - I'd miss it if you
| didn't point it out on HN.
| a2code wrote:
| If it sounds too good to be true, it probably is not.
|
| If removing weights improves some metrics, that may be a clue
| that the model is not optimal in some sense.
| kolinko wrote:
| The algorithm still uses all the weights, just not all the time
| - just skips the weights when they are not important given an
| input vector.
|
| Also, approximation methods, as a field, are not new and they
| have shown their use.
|
| Having said all that, extraordinary claims require
| extraordinary evidence - that's why I hedge the communication
| messages. It's ,,probably" until we get serious tests going on
| hatthew wrote:
| This seems similar to semi-structured (aka 2:4) sparsity, may be
| worth explicitly comparing. As far as I can tell by skimming,
| your technique:
|
| - is optimized for apple silicon - ~2x speed at 75% sparsity -
| dynamic, depends on input, applied at runtime - can choose amount
| of sparsity
|
| And 2:4 semi-structured sparsity:
|
| - is optimized for GPUs with sparse tensor cores (nvidia ampere
| and beyond) - ~2x speed at 50% sparsity - static, applied to the
| model at rest - probably worse results than your technique at 50%
| sparsity
|
| The interesting comparison I'd want to see is semi-structured
| sparsity results (50% sparsity, 2x speedup) vs your results at
| 75% sparsity (2x speedup).
___________________________________________________________________
(page generated 2024-04-17 23:00 UTC)