[HN Gopher] Sharpened Cosine Distance as an Alternative for Conv...
___________________________________________________________________
Sharpened Cosine Distance as an Alternative for Convolutions
Author : gk1
Score : 48 points
Date : 2022-07-22 14:10 UTC (8 hours ago)
(HTM) web link (www.rpisoni.dev)
(TXT) w3m dump (www.rpisoni.dev)
| DarkSucker wrote:
| Not being familiar with cosine distance, I found an NIST link:
| https://www.itl.nist.gov/div898/software/dataplot/refman2/au....
|
| It seems like the main difference between convolution and cosine
| distance is the normalization. For example, if I associate vector
| A with the windowed data set and B with the cosine difference
| "kernel", then cosine difference will have a larger magnitude
| than a convolution for small A.
|
| Cool idea. I plan to give it a whirl some day.
| stefanpie wrote:
| I remember the original twitter thread on the topic but haven't
| heard much about it since then. It would be interesting to see a
| nice arxiv submissions with some initial experiments across
| common tasks and architectures. Would also be interested in
| hardware efficient versions for fixed point data.
|
| Here is some background and more thoughts/references on this on
| this topic: https://github.com/brohrer/sharpened-cosine-
| similarity
| jschveibinz wrote:
| Interesting. However, a matched filter is the optimum detector
| wrt maximizing SNR of the output signal (the features).
|
| https://www.sciencedirect.com/topics/engineering/matched-fil...
|
| https://en.m.wikipedia.org/wiki/Matched_filter
|
| I wonder if this technique has been tested in "noisy" signal
| environments? Or is this just useful in infinite SNR
| applications?
| krapht wrote:
| optimal among linear functions, and the noise is gaussian...
|
| of course noise is not gaussian in many interesting
| applications and nothing is stopping a nonlinear function from
| getting better results than a matched filter, we just have no
| theory to discover what such a function would be
|
| so yes, like you, i'd need to see results on actual data before
| spending time to learn more about it...
| fxtentacle wrote:
| In my opinion, this looks promising:
|
| https://github.com/stevenwalton/SCS-CCT
|
| 4% improvement over Compact Transformers, which is considered
| SOTA
|
| EDIT: As meragrin_ pointed out, I read the graph the wrong
| way. It's actually 4% worse. Sorry.
| meragrin_ wrote:
| Where do you see improvement? According to the graph and
| text, there is a 4% decrease in performance. It is slightly
| worse.
| gnramires wrote:
| I tend to think of a matched filter as an "optimum denoiser"
| (in the mean squared error sense with white gaussian noise),
| but it's not an optimum detector in the general case, or an
| optimum "de-interferer".
|
| The case shown here should be more resilient to changes in
| signal amplitude and non-matching interference.
|
| Of course, because the kernel is actually learned, it can adapt
| to interference and try to be a "matched filter to what we
| want, plus anti-matched filter to interference" (i.e. give
| near-0 response to common interference signals). But SCS does
| seem like a more natural way to achieve a similarity metric and
| feature detection. (although this non-flexibility might work
| against it)
| fxtentacle wrote:
| Cosine features tend to measure if two things point in the same
| direction. Convolutional features tend to measure how much of a
| template was mixed into a signal. As such, one can expect cosine
| features to produce very similar results to a convolutional
| feature followed by a Sigmoid or Tanh. Remember what the first
| style transfer papers used for feature extraction? Conv+Tanh.
|
| After that, cosine features have been used with great success for
| generating the attention masks in transformer architectures. If I
| remember correctly, the original Transformer paper also suggested
| to normalize the features and to pre-scale the result with
| sqrt(1/N).
|
| So the new part in this formula is mostly the pow. The other
| parts (cosine distance & dividing by the norm) are already pretty
| much battle-tested in speech recognition. And the sign is just
| needed to restore what pow with an even exponent loses.
|
| In short, this suggests that applying a gamma-like transform to
| features increases contrast.
| [deleted]
| newuser4321 wrote:
| I definitely buy that there is a use case for this kernel.
| Whenever I see this kind of optimization, I feel like it's at
| least flirting with classical feature engineering. Not that
| there's necessarily anything wrong with that. You could probably
| say the same thing about Relu if you wanted. I just think the "it
| responds better to certain features" argument, which is how I'm
| understanding this, can quickly throw you back into some
| classical computer vision work where you're trying to hand
| optimize instead of relying on gradient descent to find your
| features
| feoren wrote:
| > I feel like it's at least flirting with classical feature
| engineering
|
| I, for one, would welcome more of a bridge between intentional
| human engineering and machine learning. Right now we have
| inscrutable billion-node language processing models that cause
| endless arguments about whether they actually _know_ English or
| not, but don 't actually _integrate_ well with anything humans
| are actually doing. I 'd love to see AI get mixed more fluently
| with manual solutions. It's too either-or right now; the best
| solutions will come from a mix of smart engineers working
| seamlessly with focused AI.
| freemint wrote:
| This thread seems to summarize the state of this method well:
| https://twitter.com/_clashluke/status/1497092162193793028
| Imnimo wrote:
| >it does not give the highest activation for a signal that
| matches the kernel
|
| This has the feeling of being an important property, but it's not
| clear to me whether it truly is. Do I actually care whether the
| output of my feature is extractor is "how well does the signal
| match the kernel" as opposed to some other function of the signal
| and kernel?
| karmakaze wrote:
| The Sharpened Cosine Distance animation uses a different input.
| How would it respond to the reverse pattern in the input or the
| square pulse?
| meatmanek wrote:
| Indeed. If the problem with convolution is that the rectangular
| pulse gives a stronger signal than the ramped pulses (which
| match the convolution kernel), then please show how sharpened
| cosine distance would behave differently in this problem case.
| nomel wrote:
| I can't understand why they used completely different input
| signals as a comparison.
|
| Is convolving using some sort of smooth derivative of each
| useful? Or, is that too sensitive to noise?
| frankus wrote:
| This twitter thread
| (https://twitter.com/_brohrer_/status/1232063619657093120)
| delivers.
___________________________________________________________________
(page generated 2022-07-22 23:01 UTC)