https://spectrum.ieee.org/andrew-ng-data-centric-ai

[                    ]

IEEE.orgIEEE Xplore Digital LibraryIEEE StandardsMore Sites
Sign InJoin IEEE
 
Andrew Ng: Unbiggen AI
Share
FOR THE TECHNOLOGY INSIDER
[                    ]
Explore by topic
AerospaceArtificial IntelligenceBiomedicalComputingConsumer
ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensors
TelecommunicationsTransportation
IEEE Spectrum
FOR THE TECHNOLOGY INSIDER

Topics

AerospaceArtificial IntelligenceBiomedicalComputingConsumer
ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsSensors
TelecommunicationsTransportation

Sections

FeaturesNewsOpinionCareersDIYEngineering Resources

More

Special ReportsExplainersPodcastsVideosNewslettersTop Programming
LanguagesRobots Guide

For IEEE Members

Current IssueMagazine ArchiveThe Institute

For IEEE Members

Current IssueMagazine ArchiveThe Institute

IEEE Spectrum

About UsContact UsReprints & PermissionsAdvertising

Follow IEEE Spectrum

     

Support IEEE Spectrum

IEEE Spectrum is the flagship publication of the IEEE -- the world's
largest professional organization devoted to engineering and applied
sciences. Our articles, podcasts, and infographics inform our readers
about developments in technology, engineering, and science.
Join IEEE
Subscribe
About IEEEContact & SupportAccessibilityNondiscrimination PolicyTerms
IEEE Privacy Policy
(c) Copyright 2022 IEEE -- All rights reserved. A not-for-profit
organization, IEEE is the world's largest technical professional
organization dedicated to advancing technology for the benefit of
humanity.

IEEE websites place cookies on your device to give you the best user
experience. By using our websites, you agree to the placement of
these cookies. To learn more, read our Privacy Policy.

view privacy policy accept & close
Close

Stay ahead of the latest technology trends. Become an IEEE member.

Enjoy more free content and benefits by creating an account

Saving articles to read later requires an IEEE Spectrum account

The Institute content is only available for members

Downloading full PDF issues is exclusive for IEEE Members

Access to Spectrum's Digital Edition is exclusive for IEEE Members

Following topics is a feature exclusive for IEEE Members

Adding your response to an article requires an IEEE Spectrum account

Create an account to access more content and features on IEEE
Spectrum, including the ability to save articles to read later,
download Spectrum Collections, and participate in conversations with
readers and editors. For more exclusive content and features,
consider Joining IEEE.

This article is for IEEE members only. Join the world's largest
professional organization devoted to engineering and applied sciences
and get access to all of Spectrum's articles, podcasts, and special
reports. Learn more -

Join the world's largest professional organization devoted to
engineering and applied sciences and get access to all of Spectrum's
articles, archives, PDF downloads, and other benefits. Learn more -

CREATE AN ACCOUNTSIGN IN
JOIN IEEESIGN IN
Close

Enjoy more free content and benefits by creating an account

Create an account to access more content and features on IEEE
Spectrum, including the ability to save articles to read later,
download Spectrum Collections, and participate in conversations with
readers and editors. For more exclusive content and features,
consider Joining IEEE.

CREATE AN ACCOUNTSIGN IN
Artificial Intelligence Topic Type Interview

Andrew Ng: Unbiggen AI

The AI pioneer says it's time for smart-sized, "data-centric"
solutions to big issues  

Eliza Strickland
7h
10 min read
 Andrew Ng listens during the Power of Data: Sooner Than You Think
global technology conference in Brooklyn, New York, on Wednesday,
October 30, 2019.

Andrew Ng was involved in the rise of massive deep learning models
trained on vast amounts of data, but now he's preaching small-data
solutions.

Cate Dingley/Bloomberg/Getty Images
     
deep learning Andrew Ng artificial intelligence

Andrew Ng has serious street cred in artificial intelligence. He
pioneered the use of graphics processing units (GPUs) to train deep
learning models in the late 2000s with his students at Stanford
University, cofounded Google Brain in 2011, and then served for three
years as chief scientist for Baidu, where he helped build the Chinese
tech giant's AI group. So when he says he has identified the next big
shift in artificial intelligence, people listen. And that's what he
told IEEE Spectrum in an exclusive Q&A.

Ng's current efforts are focused on his company Landing AI, which
built a platform called LandingLens to help manufacturers improve
visual inspection with computer vision. He has also become something
of an evangelist for what he calls the data-centric AI movement,
which he says can yield "small data" solutions to big issues in AI,
including model efficiency, accuracy, and bias.

Andrew Ng on...

  * What's next for really big models
  * The career advice he didn't listen to
  * Defining the data-centric AI movement
  * Synthetic data
  * Why Landing AI asks its customers to do the work


The great advances in deep learning over the past decade or so have
been powered by ever-bigger models crunching ever-bigger amounts of
data. Some people argue that that's an unsustainable trajectory. Do
you agree that it can't go on that way?

Andrew Ng: This is a big question. We've seen foundation models in
NLP [natural language processing]. I'm excited about NLP models
getting even bigger, and also about the potential of building
foundation models in computer vision. I think there's lots of signal
to still be exploited in video: We have not been able to build
foundation models yet for video because of compute bandwidth and the
cost of processing video, as opposed to tokenized text. So I think
that this engine of scaling up deep learning algorithms, which has
been running for something like 15 years now, still has steam in it.
Having said that, it only applies to certain problems, and there's a
set of other problems that need small data solutions.

When you say you want a foundation model for computer vision, what do
you mean by that?

Ng: This is a term coined by Percy Liang and some of my friends at
Stanford to refer to very large models, trained on very large data
sets, that can be tuned for specific applications. For example, GPT-3
is an example of a foundation model [for NLP]. Foundation models
offer a lot of promise as a new paradigm in developing machine
learning applications, but also challenges in terms of making sure
that they're reasonably fair and free from bias, especially if many
of us will be building on top of them.

What needs to happen for someone to build a foundation model for
video?

Ng: I think there is a scalability problem. The compute power needed
to process the large volume of images for video is significant, and I
think that's why foundation models have arisen first in NLP. Many
researchers are working on this, and I think we're seeing early signs
of such models being developed in computer vision. But I'm confident
that if a semiconductor maker gave us 10 times more processor power,
we could easily find 10 times more video to build such models for
vision.

Having said that, a lot of what's happened over the past decade is
that deep learning has happened in consumer-facing companies that
have large user bases, sometimes billions of users, and therefore
very large data sets. While that paradigm of machine learning has
driven a lot of economic value in consumer software, I find that that
recipe of scale doesn't work for other industries.

Back to top

It's funny to hear you say that, because your early work was at a
consumer-facing company with millions of users.

Ng: Over a decade ago, when I proposed starting the Google Brain
project to use Google's compute infrastructure to build very large
new networks, it was a controversial step. One very senior person
pulled me aside and warned me that starting Google Brain would be bad
for my career. I think he felt that the action couldn't just be in
scaling up, and that I should instead focus on architecture
innovation.

"In many industries where giant data sets simply don't exist, I think
the focus has to shift from big data to good data. Having 50
thoughtfully engineered examples can be sufficient to explain to the
neural network what you want it to learn."
--Andrew Ng, CEO & Founder, Landing AI

I remember when my students and I published the first NeurIPS
workshop paper advocating using CUDA, a platform for processing on
GPUs, for deep learning--a different senior person in AI sat me down
and said, "CUDA is really complicated to program. As a programming
paradigm, this seems like too much work." I did manage to convince
him; the other person I did not convince.

I expect they're both convinced now.

Ng: I think so, yes.

Over the past year as I've been speaking to people about the
data-centric AI movement, I've been getting flashbacks to when I was
speaking to people about deep learning and scalability 10 or 15 years
ago. In the past year, I've been getting the same mix of "there's
nothing new here" and "this seems like the wrong direction."

Back to top

How do you define data-centric AI, and why do you consider it a
movement?

Ng: Data-centric AI is the discipline of systematically engineering
the data needed to successfully build an AI system. For an AI system,
you have to implement some algorithm, say a neural network, in code
and then train it on your data set. The dominant paradigm over the
last decade was to download the data set while you focus on improving
the code. Thanks to that paradigm, over the last decade deep learning
networks have improved significantly, to the point where for a lot of
applications the code--the neural network architecture--is basically a
solved problem. So for many practical applications, it's now more
productive to hold the neural network architecture fixed, and instead
find ways to improve the data.

When I started speaking about this, there were many practitioners
who, completely appropriately, raised their hands and said, "Yes,
we've been doing this for 20 years." This is the time to take the
things that some individuals have been doing intuitively and make it
a systematic engineering discipline.

The data-centric AI movement is much bigger than one company or group
of researchers. My collaborators and I organized a data-centric AI
workshop at NeurIPS, and I was really delighted at the number of
authors and presenters that showed up.

You often talk about companies or institutions that have only a small
amount of data to work with. How can data-centric AI help them?

Ng: You hear a lot about vision systems built with millions of
images--I once built a face recognition system using 350 million
images. Architectures built for hundreds of millions of images don't
work with only 50 images. But it turns out, if you have 50 really
good examples, you can build something valuable, like a
defect-inspection system. In many industries where giant data sets
simply don't exist, I think the focus has to shift from big data to
good data. Having 50 thoughtfully engineered examples can be
sufficient to explain to the neural network what you want it to
learn.

When you talk about training a model with just 50 images, does that
really mean you're taking an existing model that was trained on a
very large data set and fine-tuning it? Or do you mean a brand new
model that's designed to learn only from that small data set?

Ng: Let me describe what Landing AI does. When doing visual
inspection for manufacturers, we often use our own flavor of
RetinaNet. It is a pretrained model. Having said that, the
pretraining is a small piece of the puzzle. What's a bigger piece of
the puzzle is providing tools that enable the manufacturer to pick
the right set of images [to use for fine-tuning] and label them in a
consistent way. There's a very practical problem we've seen spanning
vision, NLP, and speech, where even human annotators don't agree on
the appropriate label. For big data applications, the common response
has been: If the data is noisy, let's just get a lot of data and the
algorithm will average over it. But if you can develop tools that
flag where the data's inconsistent and give you a very targeted way
to improve the consistency of the data, that turns out to be a more
efficient way to get a high-performing system.

"Collecting more data often helps, but if you try to collect more
data for everything, that can be a very expensive activity."
--Andrew Ng

For example, if you have 10,000 images where 30 images are of one
class, and those 30 images are labeled inconsistently, one of the
things we do is build tools to draw your attention to the subset of
data that's inconsistent. So you can very quickly relabel those
images to be more consistent, and this leads to improvement in
performance.

Could this focus on high-quality data help with bias in data sets? If
you're able to curate the data more before training?

Ng: Very much so. Many researchers have pointed out that biased data
is one factor among many leading to biased systems. There have been
many thoughtful efforts to engineer the data. At the NeurIPS
workshop, Olga Russakovsky gave a really nice talk on this. At the
main NeurIPS conference, I also really enjoyed Mary Gray's
presentation, which touched on how data-centric AI is one piece of
the solution, but not the entire solution. New tools like Datasheets
for Datasets also seem like an important piece of the puzzle.

One of the powerful tools that data-centric AI gives us is the
ability to engineer a subset of the data. Imagine training a
machine-learning system and finding that its performance is okay for
most of the data set, but its performance is biased for just a subset
of the data. If you try to change the whole neural network
architecture to improve the performance on just that subset, it's
quite difficult. But if you can engineer a subset of the data you can
address the problem in a much more targeted way.

When you talk about engineering the data, what do you mean exactly?

Ng: In AI, data cleaning is important, but the way the data has been
cleaned has often been in very manual ways. In computer vision,
someone may visualize images through a Jupyter notebook and maybe
spot the problem, and maybe fix it. But I'm excited about tools that
allow you to have a very large data set, tools that draw your
attention quickly and efficiently to the subset of data where, say,
the labels are noisy. Or to quickly bring your attention to the one
class among 100 classes where it would benefit you to collect more
data. Collecting more data often helps, but if you try to collect
more data for everything, that can be a very expensive activity.

For example, I once figured out that a speech-recognition system was
performing poorly when there was car noise in the background. Knowing
that allowed me to collect more data with car noise in the
background, rather than trying to collect more data for everything,
which would have been expensive and slow.

Back to top

What about using synthetic data, is that often a good solution?

Ng: I think synthetic data is an important tool in the tool chest of
data-centric AI. At the NeurIPS workshop, Anima Anandkumar gave a
great talk that touched on synthetic data. I think there are
important uses of synthetic data that go beyond just being a
preprocessing step for increasing the data set for a learning
algorithm. I'd love to see more tools to let developers use synthetic
data generation as part of the closed loop of iterative machine
learning development.

Do you mean that synthetic data would allow you to try the model on
more data sets?

Ng: Not really. Here's an example. Let's say you're trying to detect
defects in a smartphone casing. There are many different types of
defects on smartphones. It could be a scratch, a dent, pit marks,
discoloration of the material, other types of blemishes. If you train
the model and then find through error analysis that it's doing well
overall but it's performing poorly on pit marks, then synthetic data
generation allows you to address the problem in a more targeted way.
You could generate more data just for the pit-mark category.

"In the consumer software Internet, we could train a handful of
machine-learning models to serve a billion users. In manufacturing,
you might have 10,000 manufacturers building 10,000 custom AI
models."
--Andrew Ng

Synthetic data generation is a very powerful tool, but there are many
simpler tools that I will often try first. Such as data augmentation,
improving labeling consistency, or just asking a factory to collect
more data.

Back to top

To make these issues more concrete, can you walk me through an
example? When a company approaches Landing AI and says it has a
problem with visual inspection, how do you onboard them and work
toward deployment?

Ng: When a customer approaches us we usually have a conversation
about their inspection problem and look at a few images to verify
that the problem is feasible with computer vision. Assuming it is, we
ask them to upload the data to the LandingLens platform. We often
advise them on the methodology of data-centric AI and help them label
the data.

One of the foci of Landing AI is to empower manufacturing companies
to do the machine learning work themselves. A lot of our work is
making sure the software is fast and easy to use. Through the
iterative process of machine learning development, we advise
customers on things like how to train models on the platform, when
and how to improve the labeling of data so the performance of the
model improves. Our training and software supports them all the way
through deploying the trained model to an edge device in the factory.

How do you deal with changing needs? If products change or lighting
conditions change in the factory, can the model keep up?

Ng: It varies by manufacturer. There is data drift in many contexts.
But there are some manufacturers that have been running the same
manufacturing line for 20 years now with few changes, so they don't
expect changes in the next five years. Those stable environments make
things easier. For other manufacturers, we provide tools to flag when
there's a significant data-drift issue. I find it really important to
empower manufacturing customers to correct data, retrain, and update
the model. Because if something changes and it's 3 a.m. in the United
States, I want them to be able to adapt their learning algorithm
right away to maintain operations.

In the consumer software Internet, we could train a handful of
machine-learning models to serve a billion users. In manufacturing,
you might have 10,000 manufacturers building 10,000 custom AI models.
The challenge is, how do you do that without Landing AI having to
hire 10,000 machine learning specialists?

So you're saying that to make it scale, you have to empower customers
to do a lot of the training and other work.

Ng: Yes, exactly! This is an industry-wide problem in AI, not just in
manufacturing. Look at health care. Every hospital has its own
slightly different format for electronic health records. How can
every hospital train its own custom AI model? Expecting every
hospital's IT personnel to invent new neural-network architectures is
unrealistic. The only way out of this dilemma is to build tools that
empower the customers to build their own models by giving them tools
to engineer the data and express their domain knowledge. That's what
Landing AI is executing in computer vision, and the field of AI needs
other teams to execute this in other domains.

Is there anything else you think it's important for people to
understand about the work you're doing or the data-centric AI
movement?

Ng: In the last decade, the biggest shift in AI was a shift to deep
learning. I think it's quite possible that in this decade the biggest
shift will be to data-centric AI. With the maturity of today's neural
network architectures, I think for a lot of the practical
applications the bottleneck will be whether we can efficiently get
the data we need to develop systems that work well. The data-centric
AI movement has tremendous energy and momentum across the whole
community. I hope more researchers and developers will jump in and
work on it.

Back to top

From Your Site Articles

  * Checking in with Andrew Ng at Baidu's Blooming Silicon Valley ...
    >
  * 2021's Top Stories About AI - IEEE Spectrum >
  * Andrew Ng X-Rays the AI Hype - IEEE Spectrum >

Related Articles Around the Web

  * The Man Behind the Google Brain: Andrew Ng and the Quest for the
    ... >
  * Andrew Ng - Google Scholar >
  * Andrew Ng >

deep learning Andrew Ng artificial intelligence
 
Eliza Strickland

Eliza Strickland is a senior editor at IEEE Spectrum, where she
covers AI, biomedical engineering, and other topics. She holds a
master's degree in journalism from Columbia University.

 
The Conversation (0)
Hands on the keyboard of a laptop. A colorful video game with little
creatures is on the screen.
Consumer Electronics Topic News Type

NFTs are a Grift; NFTs are the Future--Gamers, Over to You

1h
5 min read
Photo of a woman in a red top on a blue background.
The Institute Topic Type Careers Profile

GeeRemit App Moves Money and Saves It Too

08 Feb 2022
4 min read
Optometrist giving eye exam to senior patient
Topic News Type Biomedical

This AI Can Predict Heart Attack Risk From an Eye Scan

08 Feb 2022
4 min read

Related Stories

Topic Careers Type News

AI and Machine Learning Salaries Drop

 
Computing Topic Artificial Intelligence Type News

Meta Aims to Build the World's Fastest AI Supercomputer

 
Topic Artificial Intelligence Type News

AI Could Analyze Speech to Help Diagnose Alzheimer's

 
Artificial Intelligence Topic Magazine Type Feature Semiconductors

The Femtojoule Promise of Analog AI

To cut power by orders of magnitude, do your processing with analog
circuits

Geoffrey W. Burr
Abu Sebastian
Takashi Ando
Wilfried Haensch
20 Nov 2021
12 min read
     
A list of characters and letters in blue, white black and yellow.

Machine learning and artificial intelligence (AI) have already
penetrated so deeply into our life and work that you might have
forgotten what interactions with machines used to be like. We used to
ask only for precise quantitative answers to questions conveyed with
numeric keypads, spreadsheets, or programming languages: "What is the
square root of 10?" "At this rate of interest, what will be my gain
over the next five years?"

But in the past 10 years, we've become accustomed to machines that
can answer the kind of qualitative, fuzzy questions we'd only ever
asked of other people: "Will I like this movie?" "How does traffic
look today?" "Was that transaction fraudulent?"

Deep neural networks (DNNs), systems that learn how to respond to new
queries when they're trained with the right answers to very similar
queries, have enabled these new capabilities. DNNs are the primary
driver behind the rapidly growing global market for AI hardware,
software, and services, valued at US $327.5 billion this year and
expected to pass $500 billion in 2024, according to the International
Data Corporation.

Convolutional neural networks first fueled this revolution by
providing superhuman image-recognition capabilities. In the last
decade, new DNN models for natural-language processing, speech
recognition, reinforcement learning, and recommendation systems have
enabled many other commercial applications.

But it's not just the number of applications that's growing. The size
of the networks and the data they need are growing, too. DNNs are
inherently scalable--they provide more reliable answers as they get
bigger and as you train them with more data. But doing so comes at a
cost. The number of computing operations needed to train the best DNN
models grew 1 billionfold between 2010 and 2018, meaning a huge
increase in energy consumption And while each use of an
already-trained DNN model on new data--termed inference--requires much
less computing, and therefore less energy, than the training itself,
the sheer volume of such inference calculations is enormous and
increasing. If it's to continue to change people's lives, AI is going
to have to get more efficient.

We think changing from digital to analog computation might be what's
needed. Using nonvolatile memory devices and two fundamental physical
laws of electrical engineering, simple circuits can implement a
version of deep learning's most basic calculations that requires mere
thousandths of a trillionth of a joule (a femtojoule). There's a
great deal of engineering to do before this tech can take on complex
AIs, but we've already made great strides and mapped out a path
forward.

AI's Fundamental Function

Column of three yellow dots connected to a blue dot with an arrow
pointing to an output.

The most basic computation in an artificial neural network is called
multiply and accumulate. The output of artificial neurons [left,
yellow] are multiplied by the weight values connecting them to the
next neuron [center, light blue]. That neuron sums its inputs and
applies an output function. In analog AI, the multiply function is
performed by Ohm's Law, where the neuron's output voltage is
multiplied by the conductance representing the weight value. The
summation at the neuron is done by Kirchhoff's Current Law, which
simply adds all the currents entering a single node

The biggest time and energy costs in most computers occur when lots
of data has to move between external memory and computational
resources such as CPUs and GPUs. This is the "von Neumann
bottleneck," named after the classic computer architecture that
separates memory and logic. One way to greatly reduce the power
needed for deep learning is to avoid moving the data--to do the
computation out where the data is stored.

DNNs are composed of layers of artificial neurons. Each layer of
neurons drives the output of those in the next layer according to a
pair of values--the neuron's "activation" and the synaptic "weight" of
the connection to the next neuron.

Most DNN computation is made up of what are called
vector-matrix-multiply (VMM) operations--in which a vector (a
one-dimensional array of numbers) is multiplied by a two-dimensional
array. At the circuit level these are composed of many
multiply-accumulate (MAC) operations. For each downstream neuron, all
the upstream activations must be multiplied by the corresponding
weights, and these contributions are then summed.

Most useful neural networks are too large to be stored within a
processor's internal memory, so weights must be brought in from
external memory as each layer of the network is computed, each time
subjecting the calculations to the dreaded von Neumann bottleneck.
This leads digital compute hardware to favor DNNs that move fewer
weights in from memory and then aggressively reuse these weights.

A radical new approach to energy-efficient DNN hardware occurred to
us at IBM Research back in 2014. Together with other investigators,
we had been working on crossbar arrays of nonvolatile memory (NVM)
devices. Crossbar arrays are constructs where devices, memory cells
for example, are built in the vertical space between two
perpendicular sets of horizontal conductors, the so-called bitlines
and the wordlines. We realized that, with a few slight adaptations,
our memory systems would be ideal for DNN computations, particularly
those for which existing weight-reuse tricks work poorly. We refer to
this opportunity as "analog AI," although other researchers doing
similar work also use terms like "processing-in-memory" or
"compute-in-memory."

There are several varieties of NVM, and each stores data differently.
But data is retrieved from all of them by measuring the device's
resistance (or, equivalently, its inverse--conductance).
Magnetoresistive RAM (MRAM) uses electron spins, and flash memory
uses trapped charge. Resistive RAM (RRAM) devices store data by
creating and later disrupting conductive filamentary defects within a
tiny metal-insulator-metal device. Phase-change memory (PCM) uses
heat to induce rapid and reversible transitions between a
high-conductivity crystalline phase and a low-conductivity amorphous
phase.

Flash, RRAM, and PCM offer the low- and high-resistance states needed
for conventional digital data storage, plus the intermediate
resistances needed for analog AI. But only RRAM and PCM can be
readily placed in a crossbar array built in the wiring above silicon
transistors in high-performance logic, to minimize the distance
between memory and logic.

We organize these NVM memory cells in a two-dimensional array, or
"tile." Included on the tile are transistors or other devices that
control the reading and writing of the NVM devices. For memory
applications, a read voltage addressed to one row (the wordline)
creates currents proportional to the NVM's resistance that can be
detected on the columns (the bitlines) at the edge of the array,
retrieving the stored data.

To make such a tile part of a DNN, each row is driven with a voltage
for a duration that encodes the activation value of one upstream
neuron. Each NVM device along the row encodes one synaptic weight
with its conductance. The resulting read current is effectively
performing, through Ohm's Law (in this case expressed as "current
equals voltage times conductance"), the multiplication of excitation
and weight. The individual currents on each bitline then add together
according to Kirchhoff's Current Law. The charge generated by those
currents is integrated over time on a capacitor, producing the result
of the MAC operation.

These same analog in-memory summation techniques can also be
performed using flash and even SRAM cells, which can be made to store
multiple bits but not analog conductances. But we can't use Ohm's Law
for the multiplication step. Instead, we use a technique that can
accommodate the one- or two-bit dynamic range of these memory
devices. However, this technique is highly sensitive to noise, so we
at IBM have stuck to analog AI based on PCM and RRAM.

Unlike conductances, DNN weights and activations can be either
positive or negative. To implement signed weights, we use a pair of
current paths--one adding charge to the capacitor, the other
subtracting. To implement signed excitations, we allow each row of
devices to swap which of these paths it connects with, as needed.

Nonvolatile Memories for Analog AI

Layers of colored lines with red dots in the 2nd row which is blue.

 Phase-change memory's conductance is set by the transition between a
crystalline and an amorphous state in a chalcogenide glass. In
resistive RAM, conductance depends on the creation and destruction of
conductive filaments in an insulator.

Three rows of color with the middle tan layer having white dots
labeled \u201cVacancy.\u201d

In resistive RAM, conductance depends on the creation and destruction
of conductive filaments in an insulator.

Rows of colors with red dots.

Flash memory stores data as charge trapped in a "floating gate." The
presence or absence of that charge modifies conductances across the
device.

Rows of colors with plus and minus icons and a label that says \
u201cElectrochemical RAM\u201d

Electrochemical RAM acts like a miniature battery. Pulses of voltage
on a gate electrode modulate the conductance between the other two
terminals by the exchange of ions through a solid electrolyte.

With each column performing one MAC operation, the tile does an
entire vector-matrix multiplication in parallel. For a tile with
1,024 x 1,024 weights, this is 1 million MACs at once.

In systems we've designed, we expect that all these calculations can
take as little as 32 nanoseconds. Because each MAC performs a
computation equivalent to that of two digital operations (one
multiply followed by one add), performing these 1 million analog MACs
every 32 nanoseconds represents 65 trillion operations per second.

We've built tiles that manage this feat using just 36 femtojoules of
energy per operation, the equivalent of 28 trillion operations per
joule. Our latest tile designs reduce this figure to less than 10 fJ,
making them 100 times as efficient as commercially available hardware
and 10 times better than the system-level energy efficiency of the
latest custom digital accelerators, even those that aggressively
sacrifice precision for energy efficiency.

It's been important for us to make this per-tile energy efficiency
high, because a full system consumes energy on other tasks as well,
such as moving activation values and supporting digital circuitry.

There are significant challenges to overcome for this analog-AI
approach to really take off. First, deep neural networks, by
definition, have multiple layers. To cascade multiple layers, we must
process the VMM tile's output through an artificial neuron's
activation--a nonlinear function--and convey it to the next tile. The
nonlinearity could potentially be performed with analog circuits and
the results communicated in the duration form needed for the next
layer, but most networks require other operations beyond a simple
cascade of VMMs. That means we need efficient analog-to-digital
conversion (ADC) and modest amounts of parallel digital compute
between the tiles. Novel, high-efficiency ADCs can help keep these
circuits from affecting the overall efficiency too much. Recently, we
unveiled a high-performance PCM-based tile using a new kind of ADC
that helped the tile achieve better than 10 trillion operations per
watt.

A second challenge, which has to do with the behavior of NVM devices,
is more troublesome. Digital DNNs have proven accurate even when
their weights are described with fairly low-precision numbers. The
32-bit floating-point numbers that CPUs often calculate with are
overkill for DNNs, which usually work just fine and with less energy
when using 8-bit floating-point values or even 4-bit integers. This
provides hope for analog computation, so long as we can maintain a
similar precision.

Given the importance of conductance precision, writing conductance
values to NVM devices to represent weights in an analog neural
network needs to be done slowly and carefully. Compared with
traditional memories, such as SRAM and DRAM, PCM and RRAM are already
slower to program and wear out after fewer programming cycles.
Fortunately, for inference, weights don't need to be frequently
reprogrammed. So analog AI can use time-consuming write-verification
techniques to boost the precision of programming RRAM and PCM devices
without any concern about wearing the devices out.

That boost is much needed because nonvolatile memories have an
inherent level of programming noise. RRAM's conductivity depends on
the movement of just a few atoms to form filaments. PCM's
conductivity depends on the random formation of grains in the
polycrystalline material. In both, this randomness poses challenges
for writing, verifying, and reading values. Further, in most NVMs,
conductances change with temperature and with time, as the amorphous
phase structure in a PCM device drifts, or the filament in an RRAM
relaxes, or the trapped charge in a flash memory cell leaks away.

There are some ways to finesse this problem. Significant improvements
in weight programming can be obtained by using two conductance pairs.
Here, one pair holds most of the signal, while the other pair is used
to correct for programming errors on the main pair. Noise is reduced
because it gets averaged out across more devices.

We tested this approach recently in a multitile PCM-based chip, using
both one and two conductance pairs per weight. With it, we
demonstrated excellent accuracy on several DNNs, even on a recurrent
neural network, a type that's typically sensitive to weight
programming errors.

Vector-Matrix Multiplication with Analog AI

Column of colored dots connected by blue lines.

Vector-matrix multiplication (VMM) is the core of a neural network's
computing [top]; it is a collection of multiply-and-accumulate
processes. Here the activations of artificial neurons [yellow] are
multiplied by the weights of their connections [light blue] to the
next layer of neurons [green].

Rows of white square with yellow, blue and green dots around the
outside.

For analog AI, VMM is performed on a crossbar array tile [center]. At
each cross point, a nonvolatile memory cell encodes the weight as
conductance. The neurons' activations are encoded as the duration of
a voltage pulse. Ohm's Law dictates that the current along each
crossbar column is equal to this voltage times the conductance.
Capacitors [not shown] at the bottom of the tile sum up these
currents. A neural network's multiple layers are represented by
converting the output of one tile into the voltage duration pulses
needed as the input to the next tile [right].

Different techniques can help ameliorate noise in reading and drift
effects. But because drift is predictable, perhaps the simplest is to
amplify the signal during a read with a time-dependent gain that can
offset much of the error. Another approach is to use the same
techniques that have been developed to train DNNs for low-precision
digital inference. These adjust the neural-network model to match the
noise limitations of the underlying hardware.

As we mentioned, networks are becoming larger. In a digital system,
if the network doesn't fit on your accelerator, you bring in the
weights for each layer of the DNN from external memory chips. But
NVM's writing limitations make that a poor decision. Instead,
multiple analog AI chips should be ganged together, with each passing
the intermediate results of a partial network from one chip to the
next. This scheme incurs some additional communication latency and
energy, but it's far less of a penalty than moving the weights
themselves.

Until now, we've only been talking about inference--where an
already-trained neural network acts on novel data. But there are also
opportunities for analog AI to help train DNNs.

DNNs are trained using the backpropagation algorithm. This combines
the usual forward inference operation with two other important
steps--error backpropagation and weight update. Error backpropagation
is like running inference in reverse, moving from the last layer of
the network back to the first layer; weight update then combines
information from the original forward inference run with these
backpropagated errors to adjust the network weights in a way that
makes the model more accurate.

The Tiki-Taka Solution

Analog AI can reduce the power consumption of training neural
networks, but because of some inherent characteristics of the
nonvolatile memories involved, there are some complications.
Nonvolatile memories, such as phase-change memory and resistive RAM,
are inherently noisy. What's more, their behavior is asymmetric. That
is, at most points on their conductance curve, the same value of
voltage will produce a different change in conductance depending on
the voltage's polarity.

One solution we came up with, the Tiki-Taka algorithm, is a
modification to backpropagation training. Crucially, it is
significantly more robust to noise and asymmetric behavior in the NVM
conductance. This algorithm depends on RRAM devices constructed to
conduct in both directions. Each of these is initialized to their
symmetry point--the spot on their conductance curve where the
conductance increase and decrease for a given voltage are exactly
balanced. In Tiki-Taka, the symmetry-point-balanced NVM devices are
involved in weight updates to train the network. Periodically, their
conductance values are programmed onto a second set of devices, and
the training devices are returned to their natural symmetry point.
This allows the neural network to train to high accuracy, even in the
presence of noise and asymmetry that would completely disrupt the
conventional backpropagation algorithm.

The backpropagation step can be done in place on the tiles but in the
opposite manner of inferencing--applying voltages to the columns and
integrating current along rows. Weight update is then performed by
driving the rows with the original activation data from the forward
inference, while driving the columns with the error signals produced
during backpropagation.

Training involves numerous small weight increases and decreases that
must cancel out properly. That's difficult for two reasons. First,
recall that NVM devices wear out with too much programming. Second,
the same voltage pulse applied with opposite polarity to an NVM may
not change the cell's conductance by the same amount; its response is
asymmetric. But symmetric behavior is critical for backpropagation to
produce accurate networks. This is only made more challenging because
the magnitude of the conductance changes needed for training
approaches the level of inherent randomness of the materials in the
NVMs.

There are several approaches that can help here. For example, there
are various ways to aggregate weight updates across multiple training
examples, and then transfer these updates onto NVM devices
periodically during training. A novel algorithm we developed at IBM,
called Tiki-Taka, uses such techniques to train DNNs successfully
even with highly asymmetric RRAM devices. Finally, we are developing
a device called electrochemical random-access memory (ECRAM) that can
offer not just symmetric but highly linear and gradual conductance
updates.

The success of analog AI will depend on achieving high density, high
throughput, low latency, and high energy efficiency--simultaneously.
Density depends on how tightly the NVMs can be integrated into the
wiring above a chip's transistors. Energy efficiency at the level of
the tiles will be limited by the circuitry used for analog-to-digital
conversion.

But even as these factors improve and as more and more tiles are
linked together, Amdahl's Law--an argument about the limits of
parallel computing--will pose new challenges to optimizing system
energy efficiency. Previously unimportant aspects such as data
communication and the residual digital computing needed between tiles
will incur more and more of the energy budget, leading to a gap
between the peak energy efficiency of the tile itself and the
sustained energy efficiency of the overall analog-AI system. Of
course, that's a problem that eventually arises for every AI
accelerator, analog or digital.

The path forward is necessarily different from digital AI
accelerators. Digital approaches can bring precision down until
accuracy falters. But analog AI must first increase the
signal-to-noise ratio (SNR) of the internal analog modules until it
is high enough to demonstrate accuracy equivalent to that of digital
systems. Any subsequent SNR improvements can then be applied toward
increasing density and energy efficiency.

These are exciting problems to solve, and it will take the
coordinated efforts of materials scientists, device experts, circuit
designers, system architects, and DNN experts working together to
solve them. There is a strong and continued need for higher
energy-efficiency AI acceleration, and a shortage of other attractive
alternatives for delivering on this need. Given the wide variety of
potential memory devices and implementation paths, it is quite likely
that some degree of analog computation will find its way into future
AI accelerators.

This article appears in the December 2021 print issue as "Ohm's Law +
Kirchhoff's Current Law = Better AI."

From Your Site Articles

  * Startup and Academics Find Path to Powerful Analog AI - IEEE ...
    >
  * To Speed Up AI, Mix Memory and Processing - IEEE Spectrum >
  * IBM Toolkit Aims at Boosting Efficiencies in AI Chips - IEEE
    Spectrum >

Related Articles Around the Web

  * Using Analog For AI >
  * Will Analog AI Make Mythic a Unicorn? >
  * IBM Research Says Analog AI Will Be 100X More Efficient. Yes,
    100X. >

Keep Reading | Show less
{"imageShortcodeIds":[]}