[HN Gopher] On AlphaTensor's new matrix multiplication algorithms
___________________________________________________________________
On AlphaTensor's new matrix multiplication algorithms
Author : matt_d
Score : 265 points
Date : 2022-10-07 03:45 UTC (19 hours ago)
(HTM) web link (fgiesen.wordpress.com)
(TXT) w3m dump (fgiesen.wordpress.com)
| doubtfuluser wrote:
| While I agree with other commenters and also with the article,
| that this is not like ,,boom, DeepMind magically speeds up
| everything and you have to look in more detail at numerical
| stability etc depending on your use case, there is still
| something big in here: in the past algorithms were an almost
| exclusive product of long and deep thinking of experts. Now we
| saw that AI can be used for algorithm discovery. This can
| actually have quite big impact. All tiny improvements can add up,
| some improvement in sorting, some in matrix multiplication, some
| in lookups, you get the point. All that can accumulate to
| business advantages.
|
| So yes I think this is an important first result.
| mattnewport wrote:
| Proofs are a pretty important part of algorithm development.
| Proofs of correctness as well as proofs of algorithmic
| complexity bounds. At the moment that doesn't seem like
| something this type of approach can do much for, though perhaps
| it could be combined with work on automated theorem proving.
| DannyBee wrote:
| These algorithms are provably correct, and it will only
| generate provably correct algorithms (as the paper goes
| into).
| alexmolas wrote:
| but in this case, it's extremely easy to prove that the
| algorithm that they propose is correct, or am i missing
| something?
| eigenket wrote:
| "Correct" means different things in different contexts. In
| a fairly standard application of matrix multiplication you
| _don 't_ multiply matrices A and B to get AB, instead you
| have some floating point approximation to A, and some
| floating point approximation to B, and you want something
| that you can be confident approximates AB with some bounds.
| The two most important characteristics of an algorithm are
| the runtime and the error bounds you can prove.
|
| Someone smart once said getting the wrong answer in time
| O(1) is very easy.
| DannyBee wrote:
| Sorry, but this simply does not make any sense.
|
| Algorithmic correctness does not vary in different
| contexts.
|
| Algorithmic usefulness/applicability does.
|
| You are confusing the two.
|
| Correctness here means that it provably generates a
| result that meets the definition of correct matrix
| multiplication.
|
| In this case, they prove that all algorithms generated do
| (and that the system will actually only generate provably
| correct algorithms).
|
| Applicability here is whether, when applied to a
| particular {not-infinite precision computer, use case},
| it is viable to use it.
|
| That does not affect whether the algorithm is correct or
| not, only whether you can use it to achieve a particular
| result.
|
| If i have a computer with 1 bit of floating point
| precision, that does not make the algorithms all suddenly
| incorrect. Within the bounds of the what i can provide
| (not a lot), they still function exactly as they are
| supposed to. If i need 75 significant digits on this
| computer, it simply means that they are not useful for my
| computer because it cannot generate enough significant
| digits from them to be useful. That is totally orthogonal
| to whether the algorithms function as designed.
| eigenket wrote:
| An algorithm for "matrix multiplication" that assumes
| infinite precision arithmetic is used is essentially
| useless (there are some niche uses for matrices with
| entries in finite fields/rings). An algorithm for matrix
| multiplication usually comes with more than that, in
| particular some sort of error bound. Note that these
| bounds are mathematical, part of the abstract algorithm,
| and _not_ something that is a property of any particular
| implementation (your last paragraph suggests you might be
| confused by this). You want a statement that says if A '
| is close to A (in some relevant distance measure), and B'
| is close to B, then your algorithm gives something close
| to AB.
|
| The correctness you have to prove includes proving that
| your error bounds are what you say they are.
| DannyBee wrote:
| Again, useless is not the same as incorrect. It is still
| correct whether it can be used somewhere due to
| implementation limitations or not.
|
| I'm really unsure how you can possibly argue otherwise.
|
| It's like arguing that a string algorithm is incorrect
| because it doesn't run fast enough on strings to be
| usable on any current computer. It's still correct. It's
| just not usable.
|
| Unlike correctness, useless is very context specific. 100
| years from now, an infinite precision arithmetic
| algorithm may be entirely useful.
| eigenket wrote:
| I am arguing that a matrix multiplication algorithm, like
| essentially any other numerical algorithm, consists of
| (at least) the following two parts
|
| 1. The actual steps you have to follow.
|
| 2. Some form of error bounds/analysis that tell you how
| good/bad the output will be given approximate inputs.
|
| In order to prove correctness you have to prove that the
| procedure gives the error bounds you claim. The error
| bounds are something you mathematically have to prove.
| DannyBee wrote:
| "In order to prove correctness you have to prove that the
| procedure gives the error bounds you claim"
|
| You don't get to just add your own requirement for
| correctness and then force people to prove it?
|
| They claim a specific thing - they prove that thing. That
| thing suffices to prove that it succeeds at matrix
| multiplication. You for some reason really just don't
| like that as far as i can tell, and argue it doesn't
| suffice for usefulness (which i agree on)
|
| Matrix multiplication, and "essentially any other
| numerical algorithm", is not defined in terms of the
| error bounds for correctness. That is just BS. The error
| bounds depend on implementation factors, and as such,
| they are totally unrelated to correctness.
|
| Let's take a look:
| https://en.wikipedia.org/wiki/Matrix_multiplication
|
| I have read the entire definition, nowhere does it refer
| to error bounds as a requirement for successful matrix
| multiplication!
|
| The word "error" does not even appear on the page
|
| Since it's wikipedia, I also pulled out my college math
| books. Same thing.
|
| They prove correctness without any reference to error
| bounds. Those are accepted proofs.
|
| I don't see a single basic proof that has error bounds as
| part of correctness.
|
| It, again, wouldn't make any sense, because error bounds
| depend on implementation factors.
|
| So again, you simply can't add your requirement to
| correctness just because you like it. They still remain
| where they should be - usefulness for application.
| eigenket wrote:
| You seem to be confusing the mathematical definition of
| matrix multiplication with an explicit algorithm to
| compute it. The wiki page you linked is almost entirely
| about the abstract mathematical definition and talks
| about algorithms only for a single paragraph, which is
| about computational complexity. If you look at the wiki
| page for a particular algorithm (e.g. Strassen:
| https://en.wikipedia.org/wiki/Strassen_algorithm) then of
| course they talk about stability in comparison to the
| naive algorithm (although the page could be improved
| quite a bit).
|
| If your college textbooks do not mention error analysis,
| conditioning and stability then they are not numerical
| linear algebra books worthy of the name. Check out a
| reference like Trefethen and Bau's Numerical Linear
| Algebra for example. This book has a whole part (out of
| the 7 parts in the book) talking about conditioning and
| stability, and these ideas are present throughout other
| parts as well.
|
| Once more the type of analysis I'm talking about is
| emphatically _not_ implementation dependent. It is a
| property of the algorithm itself. For an example of the
| sort of statement I mean check out theorem 3.1 of this
| paper: https://arxiv.org/abs/math/0603207. If you
| disagree with me then please indicate what sort of
| "implementation factors" appear in the statement of the
| theorem.
| Analog24 wrote:
| If you look up _numerical_ algorithm in Wikiepedia you
| will find plenty of discussion of error bounds. Of course
| matrix multiplication performed on exact numbers doesn 't
| have errors, the calculation is _exact_. Computers and
| the algorithms they run do not have that luxury when
| dealing with approximations.
| adgjlsfhk1 wrote:
| There are lots of methods (e.g. LU) that are used
| millions of times a day, but don't have good error
| bounds. Most matrix algorithms used in practice have
| something resembling a proof of error bounds that
| everyone ignores because they don't do a good job of
| describing the error that you actually get when you use
| the methods.
| topaz0 wrote:
| What does "correctness" mean? It's impossible to talk
| about correctness without specifying what you are
| intending to compute. Numerical algorithms frequently
| have correctness conditions specified in terms of a
| precision (e.g. compute a floating point approximation to
| f(x) to within 1eps), and any algorithm that doesn't meet
| that precision is incorrect with respect to that
| correctness condition, by definition.
|
| So you can think of this paper as saying "suppose you
| have a correct multiplication and addition operation on
| the field of interest. Then this algorithm for
| multiplying matrices over that field, which is composed
| of a sequence of those multiply and add operations,
| computes the correct matrix product." That is a perfectly
| provable kind of fact, that as you say doesn't stop being
| the case if you switch computers or something.
|
| But fpmul and fpadd on your computer doesn't satisfy the
| condition for this proof! Therefore it doesn't apply,
| except by rough analogy. Then, other people might be
| interested in a different kind of proof, of a fact more
| like: "given two matrices of floating point numbers and
| fpmul and fpadd operations that are within .5eps
| precision, this sequence of fpmul and fpadd operations
| provably computes the matrix product such that the
| eigenvalues of the product are within .5eps of the true
| values". (edit to add: you could then also prove that the
| implementations of fpmul and fpadd on your computer
| satisfy the first condition, or replace them with
| implementations that do). That is also a well-specified
| correctness condition, and it is a correctness condition
| that is not necessarily satisfied by replacing "multiply"
| and "add" in the first algorithm with "fpmul" and
| "fpadd". This is what it means for "correctness [to be]
| context dependent". It doesn't invalidate the first
| proof, it just means we are interested in a different
| correctness condition than the first proof proves.
|
| (edit to format, and add: of course different conditions
| on precision are relevant to different applications.
| Maybe I need my matrix elements to be computed within
| some absolute error bound, but my friend needs them
| computed within a relative error bound. Or I need it to
| be able to work on subnormal numbers within a certain
| precision, but my friend only needs it to be precise for
| numbers between 1 and 2. Different algorithms may satisfy
| one condition but not the other.)
| eigenket wrote:
| Thanks for this comment. I think I started in this
| comment section trying to say something like this, but
| said it in a bad way, and then got progressively more
| confused and less clear in my responses as the day went
| on.
|
| This is much more clear than anything I would have been
| able to write.
| [deleted]
| bheadmaster wrote:
| > Someone smart once said getting the wrong answer in
| time O(1) is very easy.
|
| Reminds me of a joke about a guy at a job interview:
| "So, what kind of skills do you have?" "I can do
| mental multiplication really fast." "Ok, what's
| 102 times 376?" "87843" [enters numbers
| in calculator] "That wasn't even close to correct."
| "Yeah, but it was fast."
| topaz0 wrote:
| I haven't read the paper, but from the blog post I get
| the sense that finite fields is one of the main claims of
| usefulness for this result.
|
| I do agree that "'correct' means different things in
| different contexts."
| bee_rider wrote:
| ATLAS tried to auto-tune BLAS quite a while ago -- it works OK
| (not as good as the hand-tuned libraries with assembly kernels,
| though).
| lairv wrote:
| It will be interesting to see how well this method will perform
| on other type of algorithms. Matrix multiplication was quite
| convenient: thanks to the divide-and-conquer approach you only
| need to find algorithm for 4x4 matrices (or other small
| matrices), and there is an easy way to prove correctness of the
| algorithm
| lacker wrote:
| Is Strassen's algorithm actually used in practice? Oddly enough I
| find myself doing a lot of matrix multiplication recently. But I
| am just using cublasCgemm3mStridedBatched from Nvidia's cuBLAS
| library, and it doesn't appear to be public information how it's
| implemented. Does anyone know if it's actually using Strassen?
|
| Basically the library described at:
|
| https://developer.nvidia.com/blog/cublas-strided-batched-mat...
|
| I am a bit not-sold-yet on the AlphaTensor stuff because in
| practice it often seems like shuffling the data around in GPU
| memory is more expensive than doing the actual multiplications.
| It takes longer to move values between regular GPU memory and
| shared memory than it does to do a multiply, right? So all these
| algorithms that are optimizing the number of arithmetic
| operations, it isn't even clear to me that they're optimizing the
| right thing, because they require that you shuffle your data
| around in weird ways, and they don't generally measure the number
| of "memory moves" that are needed.
|
| That said, I would be happy to drop in a replacement for
| cublasCgemm3mStridedBatched and test out if it worked better for
| me! It doesn't seem like these new AlphaTensor matrix
| multiplication routines are available as plain old c/c++
| libraries yet, though.
| throwawaymaths wrote:
| Why would you just use a warp and do the memory moves on the
| GPU?
| lacker wrote:
| That is correct, if I understand you correctly, but that
| doesn't solve the entire optimization problem. You still have
| to figure out how exactly to handle tiles and transfers to
| shared memory. This might be a good page for answering this
| question:
|
| https://docs.nvidia.com/deeplearning/performance/dl-
| performa...
| [deleted]
| lordnacho wrote:
| Isn't there some heuristic that uses advanced multiplication
| algorithms if the matrix is big enough? Perhaps also checking
| for sparseness if that matters.
| bee_rider wrote:
| IIRC, Strassen's algorithm is less stable, so it isn't just a
| "if the matrix is big enough, go for it" sort of thing,
| necessarily... although, I've never looked into exactly where
| the issue shows up.
|
| I wonder how well it parallelizes.
| defrost wrote:
| It's useful enough for the relatively small class of people
| dealing with matrices of sizes 8192x8192 and above, moreso to
| those writing backend libraries for targeted computation
| architectures that can utilise the various weights for moving
| blocks of data from input source streams between computation
| nodes, etc.
|
| You're correct - the gains come from really knowing the
| computational architecture and using this _approach_ to find
| tweaks that optimise operations .. where those operations aren
| 't just atomic mults and adds, but include piped multiply-adds
| and data moves.
| rpep wrote:
| There're many people doing larger matrices, but of course
| they're mostly sparse and so this isn't relevant to them.
| [deleted]
| verdverm wrote:
| I wonder, was/is the ai able to reproduce the best know
| algorithms we already have? Can it do so reliably?
|
| If this Algo is only good for large matrices, can it reproduce
| the one for small matrices?
| naillo wrote:
| Some good critique on this result:
| https://twitter.com/cHHillee/status/1577713102434361344
|
| TLDR seems like the baseline strassen implementation they used is
| questionable wrt how really optimal it is in the first place.
| topaz0 wrote:
| Also note: a few tweets down in this thread, the tweeter says
| "you're right, I retract my criticism". So maybe not extremely
| well thought out critique after all. On twitter of all places.
| ncmncm wrote:
| Possibly sorting networks could get practical benefit from the
| Alpha treatment.
|
| I think the largest sorting network proven optimal is for a block
| size in the teens (13?).
|
| Analogously to matrix multiplication, it matters most where
| comparisons are much more expensive than swaps.
| jackmott wrote:
| zekrioca wrote:
| Always caution someone who wants to follow in this research: When
| somebody promotes a fast algorithm there is often a catch. In
| this case the issue is numerical stability. There is a theorem
| that states that any algorithm for n-by-n matrix-matrix
| multiplication that is componentwise forward stable (as good as
| it gets in this situation) much necessarily use n^3 scalar
| multiplications. The authors will therefore waste their time if
| they carry out their plans and try to optimize for stability. The
| standard algorithm has the nice property and no faster algorithm
| can have this property. The question of fast matrix
| multiplication was raised recently on mathoverflow.net, see
| https://mathoverflow.net/q/421304/110176 and the answers given
| there.
| KKKKkkkk1 wrote:
| We're talking about algorithms for TPUs, which quietly quantize
| your float32 matrices to bfloat16 behind your back [0]. This is
| aimed at a crowd that doesn't care about stability.
|
| [0] https://cloud.google.com/blog/products/ai-machine-
| learning/b...
| alevskaya wrote:
| There's a big difference between not caring about stability,
| and being willing to trade precision for better memory
| bandwidth for an application that doesn't benefit from
| increased precision. When doing large training jobs on TPUs,
| stability is paramount! It's true that you have to know more
| about what you're doing when you reduce bit-depth - the
| horrors of floating point are harder to ignore, and it's
| wildly inappropriate for many scientific computations.
| However the reduction of bit-depth is likely to continue as
| we seek to make modern models more efficient and economical
| to train and use.
| bjourne wrote:
| What does this mean in practice? For ML, we usually don't
| care if a weight is 0.05 or 0.10 cause we have millions of
| weights. We do care if one 1.237e+27 instead of 1.237e-3
| though.
| Der_Einzige wrote:
| Weight changes in one neuron can have dramatic and non
| linear or obviously predictable impact on the performance
| of a full model.
| yarky wrote:
| When building a mcmc sampler I was too lazy to properly
| code a matrix approximation needed to avoid some
| mathematical black hole and the corresponding underflow.
| It was cheaper to just ignore the faulty simulations.
|
| Turns out our results were better than the papers we
| compared to, both in time and precision.
|
| I am not that familiar with ml, but can't you just ignore
| those faulty weights?
| psb217 wrote:
| With MCMC, depending on application, it seems risky to
| just toss out the NaN/inf results. I'd guess these
| numerical issues are more likely to occur in certain
| regions of the state space you're sampling from, so your
| resulting sample could end up a bit biased. In some cases
| the bias may be small or otherwise unimportant, so the
| speed-up and simpler code of filtering NaN/inf results is
| worth it, but in other cases (like when the MCMC samples
| feed into some chain of downstream computations) the bias
| may have sneaky insidious effects.
| yarky wrote:
| I didn't think deeply about this back then since my
| parameter estimates where close/better than the
| literature I compared to, but now I'm interested in
| checking the distribution of those NaN/inf. If I recall
| correctly they were uniformly distributed throughout an
| adaptive phase.
| mxkopy wrote:
| When people talk about AI taking over the world, a funny
| image pops up in my head where a robot is trying to enter
| a frying pan. When you ask it why it's doing that, it
| says "because I feel like [NaN, NaN, 2.45e24, NaN]",
| which is a perfectly valid reason.
|
| I'm not at all caught up with the this side of ML but my
| first instinct is that faulty weights would lead to
| interpretability issues. The numbers represented by
| NaN/Inf vastly outnumber the ones within precision range,
| so interpreting them is much more of a guess.
| j7f3 wrote:
| in numerical analysis 101 you learn not to use algorithms
| that don't have certain properties and numerical
| stability is one of them
|
| what good will it do to compute something if its error is
| unbound?
|
| the issue of the accumulation of roundoff errors is
| generally speaking unavoidable when it's linear but
| fortunately they tend to be small
| adrianN wrote:
| Numerical errors have the annoying tendency to accumulate
| if you're not careful. So doing one matrix operation with
| low precision might be okay, while doing a dozen might
| completely garble your result.
| omegalulw wrote:
| This is not that relevant for ML. Each gradient pass will
| re-compute your cost function and the gradients so errors
| are not likely to accumulate. The main thing is to not
| make errors big enough that you end up in a completely
| different part of the parameter space derailing progress
| which is what the above commenter points out.
| emn13 wrote:
| This isn't true in general. Very specific ML algorithms
| that were likely developed with years of blood and sweat
| and tears may have this kind of resiliency, but I've been
| in the the numerical weeds enough here that I wouldn't
| bet on even that without a real expert weighing in on it
| - and I wonder what the tradeoff is if it's true there.
| It's very easy to have numerical stability issues
| absolutely crater ML results; been there, done that.
|
| I have some ~15 year old experience with the math behind
| some of this, but actually none with day-to-day deep
| learning applications using any of the now-conventional
| algorithms, so my perspective here is perhaps not that of
| the most pragmatic user. The status quo may have
| improved, at least de facto.
| thesz wrote:
| It is extremely relevant for ML.
|
| I am familiarizing myself with recurrent neural networks
| and getting them trained online is a pain - I get NaNs
| all the time except for very small learning rates that
| actually prevent my networks to learn anything.
|
| The deeper network is, the more pronounced accumulation
| of errors in online training is. Add 20-30 fully
| connected (not highway or residual) layers before softmax
| and you'll see wonders there, you won't be able to have
| anything stable.
| kelseyfrog wrote:
| I'm not really sure there is evidence for that. In fact,
| depending on your interpretation of why posits[1] work,
| we may even have empirical evidence that the opposite is
| true.
|
| 1. https://spectrum.ieee.org/floating-point-numbers-
| posits-proc...
| zekrioca wrote:
| My concern is that there are not enough people who are
| qualified to determine if a fast algorithm can be used or
| not. It feels reckless to include less stable algorithms in a
| general purpose library when the vast majority of users are
| mainly concerned with speed and blissfully unaware of the
| pitfalls of floating-point arithmetic.
| MatteoFrigo wrote:
| My reading of the paper is that the new 4x4 algorithm only
| works in Z/(2), where there are no issues of roundoff
| errors. (Z/(2) is the field of integers modulo 2.) The
| paper seems to say that for real numbers, Strassen is still
| the best known algorithm for the 4x4 case.
|
| (Disclaimer: googler, I have nothing to do with this
| research.)
| WithinReason wrote:
| If there are in fact stability issues, I wonder which is
| cheaper: using this fancy algorithm or changing bfloat16 to
| something like bfloat14 and using a more stable matmul.
| zasdffaa wrote:
| I don't see how that can be true. Lack of precision is one
| thing, lack of stability is very different.
|
| Instability leads to divergence from the true answer, and I
| would expect it to mean super-linear divergence (though I am
| not an expert in this) which would quickly destroy any
| meaningful result (=> chaotic behaviour). But I'm not an
| expert.
| adgjlsfhk1 wrote:
| In practice this doesn't happen because numerically
| unstable NNs tend to have bad loss. A simple way to see
| this is that instability means that the network is highly
| sensitive to the inputs, which means that the network will
| give wildly different results for basically the same input
| which is wrong. Furthermore, if the weights of your NN are
| such that you are getting dramatic overflow/underflow that
| prevents it from correctly predicting, that will have a
| high loss, and the process of training will move towards
| parameters that don't have these rounding errors blow up.
| johndfsgdgdfg wrote:
| I am not a fan of Google. But this is such a bizzare,
| deliberately misleading, snarky comment. Lower precision
| floating point arithmetic is very common in ML training.
| There's no 'behind your back' going on here.
| why_only_15 wrote:
| bfloat16 is stable enough for ML training, which is what TPUs
| exist for
| pfortuny wrote:
| Be careful: 16 bits was quite a lot of colors around 20
| years ago. Now we would laugh at it. Ditto for 640Kb of RAM
| (who would need more?) etc.
|
| Not trying to be dismissive just saying that...
| computational limits are limits on what can be done, in the
| end.
| mjan22640 wrote:
| 30 years ago. The time flies...
| pfortuny wrote:
| well, you are right and I grow old, I grow old, I shall
| wear the bottoms of my trousers rolled...
|
| I stand corrected.
| omegalulw wrote:
| Wrong way to think about the problem. Lowering bits does
| not mean lowering model capacity. Its the opposite in
| fact - it allows you to you to fit more parameters.
| pfortuny wrote:
| Well, yes _within your constraints_. In the end, you are
| choosing between two aspects. Same as with screens: you
| could have resolution (1024x768!!) or color (16bits!).
|
| Edit: the term I could not remember is tradeoff.
| adgjlsfhk1 wrote:
| kind of. if we get computers that are 1000x faster, it
| just becomes a tradeoff between higher precision or 1000x
| more parameters. the reason resolution has stopped being
| pushed is that our eyes have severe diminishing returns.
| it's not yet known whether brains do.
| rubatuga wrote:
| Not always though!
| chrisshroba wrote:
| Excuse my lack of knowledge here, but what is stability?
| zekrioca wrote:
| Numerical stability is when you do operations on the matrix
| while still keeping an eye on the overflow and underflow
| flags, all the way during the algorithm execution.
|
| This means regardless of how big your matrix is, or how big
| or small your numbers are -- or even the relation between
| them --, you algorithm is going to be stable and accurate. If
| it can't be (stable), the library must let you know. This is
| so you can keep the numbers scaled such that they are not too
| large, nor too small in order to keep them stable for the
| operations you need to execute.
| joppy wrote:
| When working over a finite field there are no stability issues
| though, since arithmetic is exact.
| zekrioca wrote:
| It will always depend on the application.
| hackpert wrote:
| While your point about numerical stability is correct in
| general, there are no numerical stability issues here and I
| think this conception, which I've seen in more than one place
| now, stems from a fundamental misunderstanding of the paper's
| results. While they _did_ come up with a faster TPU/GPU
| algorithm too, the primary result is not a fast matmul
| approximation, it is an exact algorithm comprising of stepwise
| addition/multiplication operations, and hence is numerically
| stable and should work for any ring
| (https://ncatlab.org/nlab/show/ring). AlphaTensor itself does
| not do the matrix multiplication, it was used to perform an
| (efficiently pruned) tree search over the space of operations
| to find an efficient, stable algorithm.
| zekrioca wrote:
| Directly from the paper's "Discussion" section:
|
| > One important strength of AlphaTensor is its flexibility to
| support complex stochastic and non-differentiable rewards
| (from the tensor rank to practical efficiency on specific
| hardware), in addition to finding algorithms for custom
| operations in a wide variety of spaces (such as finite
| fields). We believe this will spur applications of
| AlphaTensor towards designing algorithms that optimize
| metrics that we did not consider here, such as numerical
| stability or energy usage.
| hackpert wrote:
| Right, but doesn't that mean that it could potentially be
| used for designing algorithms that have componentwise
| numerical stability over some kind of floating point
| standard, but this, by definition being a result over
| finite fields, should be numerically stable?
|
| (apologies if I misunderstood, I wasn't calling you out
| specifically but a generalized misconception I've noticed
| in a lot of other discussions so far)
| ekelsen wrote:
| In practice you don't need to recurse all the way down. 1 level
| of Strassen is enough to get real speeedups (if speedups are
| possible at all) and certainly for deep learning, the
| instability introduced by a single level will not matter.
| zekrioca wrote:
| Do we need forward stability in the componentwise relative
| sense? This very much depends on the underlying real life
| application. If we ignore the question of accuracy and
| stability, then we just might endanger the people who depend
| on our software.
| benreesman wrote:
| They're obviously talking about bfloat on TPU.
|
| The day when a lot of wrong math adds up to a computer drawing
| a pretty picture. Who would have thought.
___________________________________________________________________
(page generated 2022-10-07 23:02 UTC)