[HN Gopher] Promise of Analog AI Feeds Neural Net Hardware Pipeline
___________________________________________________________________
Promise of Analog AI Feeds Neural Net Hardware Pipeline
Author : rbanffy
Score : 45 points
Date : 2021-12-23 14:02 UTC (8 hours ago)
(HTM) web link (spectrum.ieee.org)
(TXT) w3m dump (spectrum.ieee.org)
| mrfusion wrote:
| Has anyone figured out how to do a neural network with
| backpropagation all in analog hardware? That would be
| fascinating.
| plutonorm wrote:
| Pattern Recognition in a Bucket
| https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97...
| spiritplumber wrote:
| Maybe letting the universe in a little bit like that, will help.
| huachimingo wrote:
| "The best technology is that who doesnt exists, because then you
| dont have any problems left".
| dragontamer wrote:
| Analog computers (aka: Op-amps) are incredibly cool. But there's
| one major, major problem with them: Digital electronics introduce
| a severe amount of noise, lowering their accuracy and
| applicability.
|
| Its not hard to make a fast analog computer out of transistors
| (Indeed: transistors are innately analog devices). What is hard,
| is to have that analog computer work with a digital computer on
| the same power network... especially if you want the analog
| computer's noise floor to be usable. The noise levels are just
| too much.
|
| --------
|
| Because your values are stored as the physical voltage on various
| devices, you may write 1.05V... but maybe when you read it, its
| 1.04V due to digital noise (~1% error in this case). Is that
| acceptable?
|
| 8-bits is just 2-digits of accuracy. 16-bits is ~5 digits, still
| possible. 32-bits (12 digits of accuracy) is a pipedream.
|
| However, programmers have been lowering the accuracy of neural
| net programs / computers lower and lower. Apparently, its a field
| of compute where accuracy is just not that important. 2 or 5
| digits of accuracy (8-bit compute or 16-bit compute) is
| acceptable, with so many Tensorflow programs using BFloat16 units
| or smaller.
|
| As such, maybe neural nets can be done in analog.
| naruvimama wrote:
| Noise in a complex analog circuit is generally a problem
| because it propagates uncorrected.
|
| A ReLU for instance acts as a quantiser and so there are local
| checks.
| netizen-936824 wrote:
| Is there any reason that a device which smooths signal could
| not be introduced in the circuit, separating the analog
| computer from the rest of the digital electronics on the same
| circuit.
|
| Electronics is not my studied field so I may be missing some
| things, but this seems like a rather simple problem for an EE
| to handle, no?
| dragontamer wrote:
| > Is there any reason that a device which smooths signal
| could not be introduced in the circuit, separating the analog
| computer from the rest of the digital electronics on the same
| circuit.
|
| When you "smooth out 4GHz signals", it means you ignore 4GHz
| signals.
|
| Which means you no longer can operate at 4GHz. Instead, you
| operate at 1GHz (or slower). Any form of "smoothing"
| literally cuts your processing rate down, and now the digital
| circuit is just way faster than you.
|
| --------
|
| Analog circuits smooth out 60Hz (power-line noise), and
| 30,000 Hz (VRM noise / Power-supply noise) all the time and
| try to operate at higher frequencies where those noises are
| non-existant.
|
| In these cases, you can use a "high pass filter" (smooth out
| low-frequencies)... which is roughly a memory-limiter. (If
| you forget everything every 0.008 seconds, then 60Hz noise
| (aka: noise that only occurs every 0.016 seconds) will be
| forgotten.
|
| --------
|
| The low-pass filter (smooth out the high-frequencies) is the
| opposite, its an averaging / smoothing filter that you're
| more familiar with. If you average together all values every
| 0.01 seconds, you'll not be able to see 1000Hz information
| (aka: you have a 100Hz low-pass filter. High frequencies are
| averaged away, so the noise in the 1000Hz or 10,000Hz band is
| lost).
|
| But it also means all information in the 1000Hz or 10,000Hz+
| bands are also lost.
|
| ------
|
| Not that I ever dealt with GHz level circuits mind you. I'm
| just electronic hobbyist, who deals with MHz level issues.
| I'm sure GHz levels have all sorts of weird problems.
|
| But analog devices are quite fun to use and design. They're
| incredible... cheap and effective, no coding required. I do
| recommend people to play with Op-Amps and get your
| derivatives, integrals, multipliers, logarithms, addition and
| subtraction circuits all set up. You can do a lot of math at
| near instantaneous speeds (nanoseconds level delay) all
| analog.
|
| An Op-Amp with 800MHz gain/bandwidth (such as LTC6228:
| https://www.analog.com/media/en/technical-
| documentation/data...) can probably operate over 100MHz
| signals easily. Or in digital terms: that's a "computer" that
| can calculate (derivative/integral/multiplier/logarithm/addit
| ion/subtraction) every 10-nanoseconds... roughly the speed of
| a modern L2 cache lookup.
|
| That's fast. And "OpAmps" are kind of the easy-mode circuit.
|
| Go into "hard mode" and do things with BJTs directly, and
| you're at 80GHz bandwidth-gain.
| (https://www.infineon.com/dgdl/Infineon-BFP840FESD-
| DS-v02_00-...)... maybe practically usable at 10GHz or so
| (faster than any digital computer)
| ttqidebg wrote:
| Using more parameters you can average out the errors and
| recover some accuracy.
| dragontamer wrote:
| Digital errors are due to the clock. They pull the voltage
| down on each clock tick. Then all the circuits use power, and
| then everything settled until the start of the next clock
| tick.
|
| This error is high frequency with a huge number of harmonics
| (square wave), meaning it transmits as a radio wave very
| effectively, propagating the error to be correlated on many
| different analog parts.
| ttqidebg wrote:
| Sure, but it's effectively random which way the model
| learns the parameter impact and how values are used by
| subsequent layers. I'm assuming layers in the NN. Adding
| noise with things like dropout encourages the model to
| 'grow' redundancy.
| joe_the_user wrote:
| The problem I'd see is that the raw randomness of
| electronics is not the nice, uniform distribution that
| indeed _may_ help NNs learns. It can be, often is,
| biased, skewed, time-varying and just about anything
| else. It could vary from one part of the chip to another,
| one chip to another and so-forth. The various circuits
| exists to turn a wide range of noise into digital ons and
| offs. Which means the raw circuits can have this wide
| range unless controlled.
|
| It's worth considering that no one uses raw electronic
| noise for random/pseudo-random number generators. And
| that's an application that's in demand.
| dragontamer wrote:
| Hmmm... I see you're coming in from another angle but its
| also related to noise.
|
| -------
|
| I guess I was relating analog-noise to quantization noise
| of the FP16 format. For example, the number 4098 cannot
| be represented in FP16, you round that number to 4096 or
| 4100.
|
| This is seen as "noise" perhaps in your perspective, but
| I'm seeing it as "loss of precision". From my
| perspective, neural nets are clearly able to work even
| with low-precision (ie: numbers like 4098 being rounded
| to 4096 in every step of every calculation). "Noise" in
| my model is this "number shifting" effect.
|
| So from my side of things: neural nets seem to work with
| 8-bits or 16-bits of precision. And analog circuits are
| able to achieve that level of precision in practice
| (though its called 2-digits of precision or 5-digits of
| precision in the analog-world, maybe SNR ratio of 24db or
| 48db).
|
| ---------
|
| But from your perspective, noisy inputs are themselves a
| strategy that people use to train neural nets with.
| Literally adding noise and then compensating with a
| larger model / more training.
|
| Maybe its all noise... one artificial, the other caused
| by physical limitation (quantization noise is... noise
| after all), but it was a bit confusing for a post or two
| because I think we were coming in from different
| perspectives for a second.
| ttqidebg wrote:
| It is counter intuitive. Noise is often an essential part
| to training, especially when it comes to generalization.
| Sometimes there is enough noise in the training data to
| be sufficient.
|
| Controlling how much noise and where it goes so that you
| end up with something usable is still a bit of an art.
| Not just in designing the model but also the training and
| distillation process. It's a pain in the ass, but if the
| perf is there it can be worth it.
|
| I'm inference speed bottlenecked so I'm keeping an eye on
| this sort of tech.
| dragontamer wrote:
| I've done enough neural net stuff to know that noise can
| knock you out of local minima / local maxima. Well, not
| even neural nets... machine learning in general benefits.
|
| But there's a difference between noise that you can
| control (ie: on during training and you can
| increase/decrease at will) and noise that's fundamental
| to the system and uncontrollable.
|
| Still, since all neural nets show a resilience to a
| degree of noise, maybe that makes analog-circuits
| suddenly viable.
___________________________________________________________________
(page generated 2021-12-23 23:01 UTC)