[HN Gopher] Jim Keller on AI, RISC-V, Tenstorrent's Move to Edge IP
___________________________________________________________________
Jim Keller on AI, RISC-V, Tenstorrent's Move to Edge IP
Author : JoachimS
Score : 136 points
Date : 2023-06-13 13:44 UTC (9 hours ago)
(HTM) web link (www.eetimes.com)
(TXT) w3m dump (www.eetimes.com)
| DustinBrett wrote:
| RISC architecture is going to change everything
| frozenport wrote:
| Kinda disappointed that Tenstorrent is pushing one of the world's
| most boring architectures.
|
| Its a regular chip with a few extra units bolted on the end,
| thats been copy and pasted in a grid.
|
| We've seen the same stuff from folks like Meta, even Tesla.
| mk_stjames wrote:
| I noted how, when they first started, they talked mostly about
| developing 'AI accelerators' and it felt mostly like they were
| talking about big, GPGPU style chips to go head to head with
| nVidia. Thousands of small SIMD cores doing matrix multiplies,
| with fast memory and pcie. Maybe something halfway between
| Cerebras size and and Nvidia Hopper. A tall order but something
| really needed.
|
| Then at some point it feels like Jim got hooked on the idea of
| RISC-V everything, and they pivoted their messaging to talking
| about these more CPU-like chips with a main R-V64 8-wide
| decoding, state of the art OoO execution, etc. That sounds more
| like a RISC-V competitor to AMD Zen instead of a competitor to
| an nVidia GPU.
|
| And they they talk about that just being the interface to the
| AI chip later but... it really feels like that saw 'hey we can
| get all this RISC-V stuff for free essentially, and really take
| over the development of the spec, and that is easier than
| figuring out how to develop an GP AI chip and a stack that
| competes with CUDA to go with it, so that is easier to
| start...'
|
| I'm totally a non-expert though and the preceding is just what
| I've picked up from watching interviews with Jim (who I just
| find awesome to listen to).
| blihp wrote:
| In the interview he runs down the issues they encountered
| going down the pure AI accelerator path. It sounds like
| they've decided the opportunity wasn't there (i.e. too hard)
| so they've pivoted.
| mtkhaos wrote:
| Logic is simple inline with reducing power draw from a
| simplistic instruction set.
|
| Move into a space where we have rapid manufacturing for
| Specialized chips. Alongside the concept inherit to Nvidia's
| DPU and you have something very Interesting.
| kevin_thibedeau wrote:
| It makes more business sense to have more general purpose
| hardware that can be pivoted to other applications. Lots of
| AI ASIC vendors are going to go belly up in the coming years
| as their platforms fail to attract customers. Carving out a
| tiny niche with limited demand and no IP moat is very risky
| in the IC world.
| panick21_ wrote:
| Fast CPU performance is necessary for AI workloads too. You
| need a fast CPU and combine that with lots of Vector or
| Tensor processing. Lots of applications need both. They have
| done both for a while.
| DesiLurker wrote:
| sounds like a variation of that old MIT startup called Tilera,
| very lightweight CPUs with high performance interconnect
| fabric. at that time I remember thinking it was a solution
| looking for a problem.
|
| IDK whats wrong with that architecture for AI/ML but I feel too
| much overhead in full on CPUs. I guess thats where lightweight
| risc cores come in. Personally what I'd like to see is a clever
| architecture utilizing a grid architecture with a stack based
| process for communicating with local nodes using a bare minimal
| language like forth so extremely light 'nodes' to do matrix
| math & nothing else.
| baybal2 wrote:
| [dead]
| my123 wrote:
| Tilera itself is owned by NVIDIA nowadays.
| ftxbro wrote:
| I think boring can be good, I'm not an expert but 'extra units
| copy and pasted in a grid' is exactly how I imagine a hardware
| AI accelerator to be like
| zackmorris wrote:
| I agree, and I've wanted a grid of 1000+ cores for 25 years
| now, once I realized that the only bottleneck in computing is
| the bus between CPU and memory back in the late 90s. The only
| chip that comes even close to what I want for a reasonable
| price is Apple's M1 line, but they added GPU and AI cores
| which defeat the purpose.
|
| The churn with GPUs and now AI cores is too much for me. I
| just can't waste time manually annotating the code I want to
| run concurrently, on GPU, on AI core, whatever. To me, it
| looks like everyone is putting 10-100 times more work into
| their code than what should be needed. I see the same pattern
| repeated with web development and the explosion of walled
| garden platforms for mobile and smart devices. So much work
| for so long for so few results.
|
| Just give me a big dumb boring grid of cores and a self-
| parallelizing language to program them. Stuff like
| Julia/Clojure/MATLAB/Erlang/Go come close, but each have
| poison pills that make reaching the mainstream untenable.
| Someday I want to write a language that does what I need, but
| that day will likely never come, because every day is Quantum
| Leap for me, I just go to work to make rent, pushing the rock
| up that hill like Sisyphus, only to watch it roll down and
| have to start all over again. Inadequate tooling has come to
| totally dominate every aspect of my work, but there may never
| be time to write better stuff.
| frozenport wrote:
| This is Cerebras, and the cost is wasted compute and
| inaccessible price point
| MisterTea wrote:
| We need operating systems designed to make the resources
| easily accessible across a network. What we are running
| today are mainframe operating systems where one computer
| does all the work for concurrent users.
|
| Using plan 9 has taught me that we are far from done
| designing computers and operating systems - we're trying to
| build the future on obsolete machines running obsolete
| operating systems.and it's not going well given all the
| ugly mutually incompatible nonsense taped and bolted on to
| hide their age.
| ftxbro wrote:
| > "Stuff like Julia/Clojure/MATLAB/Erlang/Go come close,
| but each have poison pills that make reaching the
| mainstream untenable."
|
| they have 'mojo' now but it might be proprietary like
| matlab or mathematica
| snvzz wrote:
| >poison pills
|
| Unless explained, it sounds like FUD to me.
| adgjlsfhk1 wrote:
| What is the poison pill with Julia for you?
| comboy wrote:
| I think you are oversimplifying. You've chosen arbitrary
| abstraction layer. It's like saying it's just transistors
| again. Or it's just machine learning. It matters what they can
| deliver and Jim's track record is best in the whole industry. I
| think they have a pretty good understanding what's the best
| approach to bring outstanding results given technology
| available plus I think currently they want to deliver something
| to get connected with decent clients and then be able to
| optimize for actual real world use cases.
|
| I'd love to see chips made out of millions of small computing
| blocks, neuron alike, without a common clock, with local
| memory, maybe even with some analog electronics involved. But
| I'm pretty sure people who are actually working on this kind of
| stuff could provide me a list of reasons why it's silly (at
| least given current technology limitations).
| frozenport wrote:
| Lol.
|
| If they knew they wouldn't be pivoting.
|
| Jim doesn't come out a technical person, he seems mostly
| about realizing other peoples ideas or IP.
|
| Would have been great to see him leading something
| revolutionary.
| sva_ wrote:
| I liked the part in which he pointed out that on the Linux
| kernel, he could get a fix within an hour, while on NT he had to
| wait a year. His general emphasis on open source is very
| encouraging. I didn't know he came around like that.
| klelatti wrote:
| > "My belief is in the next 5 to 10 years, RISC-V will take over
| all the data centers," Keller told EE Times,
|
| Isn't this essentially impossible?
| snvzz wrote:
| >Isn't this essentially impossible?
|
| RISC-V is inevitable.
| cjsplat wrote:
| Cloud services have a lot of back end.
|
| Cluster management, file systems, disk / storage systems,
| network management, database systems.
|
| None of those require user or OS instruction set compatibility
| for legacy apps that are hard or impossible to recompile.
|
| And most of these applications don't really require gonzo
| superscalar performance. Add more cores, support more data
| streams.
|
| If you can eliminate licensing costs for that portion of your
| fleet, then you only need to expand the ISA compatible portion
| of your fleet as demanded by paying customers.
|
| As an example, suppose all of a cloud provider's services can
| migrate to RISC-V. As organic demand for x86 Cloud among
| customers grows, services can shift incrementally to the
| cheaper home grown platforms. And since the freed up machines
| are at least partially depreciated, the cost of these servers
| is much less than what a customer would pay for new servers on
| prem. (depreciated Cap-ex, far better Op-Ex).
|
| The interesting question is the transition rate of end customer
| apps to the new ISA vs the growth rate of locked ISA apps.
|
| Eventually the locked ISA apps portion becomes a lot like the
| current IBM mainframe business. Very valuable to a very small
| number of customers.
|
| The only counter for this is if x86 can crank performance per
| $TCO so far that the non-x86 branch can't compete in business
| terms, which has historically been the issue with ARM.
| als0 wrote:
| > The only counter for this is if x86 can crank performance
| per $TCO so far that the non-x86 branch can't compete in
| business terms, which has historically been the issue with
| ARM.
|
| If we take AWS for example, isn't the performance per TCO
| better of an Arm-based Graviton instance better than x86? I
| don't think the historical issue you cite represents the
| future.
| cjsplat wrote:
| Impossible to know from the outside.
|
| We know what they are selling it for, but that isn't the
| same.
|
| True TCO needs to include the cost to develop the chip -
| after all, that is folded into the x86 price.
|
| If you assume that the Graviton project is $250M per chip
| design for the 3 iterations, and the online estimates of 1
| million chips is accurate, then you need to add about $750
| per CPU, beyond the probably $250 per chip fab'ed and
| packaged.
|
| $1000 per chip gets you a lot of x86 horsepower.
| doctor_eval wrote:
| I read this and immediately thought of that aphorism, which I
| heard back in the dialup days when it seemed the internet was
| never going to take off:
|
| Important changes always take much longer than we expect, but
| have much greater impact than we imagine.
| twoodfin wrote:
| My optimistic take: "All the data centers" are the Cloud
| hyperscalers, who are increasingly delivering value through
| PaaS/SaaS vs. raw VMs and IaaS.
|
| They're choosing the CPUs they like best, can turn over quickly
| if it's worthwhile, and if the performance/economics of RISC-V
| are suitably appealing will do so.
|
| I wonder how much of, say, S3's infrastructure is running on
| Graviton?
| guerby wrote:
| I hope RISC-V servers will come with open BIOS, given Ron
| Minnich stance on the proprietary BIOS issue during his time
| at google I think hyperscalers would like that too.
|
| Recently had basic BIOS/BMC bugs it's annoying as hell.
| jylam wrote:
| Why ?
| klelatti wrote:
| My interpretation of 'take over' would be a majority of new
| server installs would be RISC-V based. There is a lead time
| for development, orders etc plus customers have to be content
| to switch to a new architecture. Amazon's Arm program started
| what 6ish years ago and they are at 20% installs (from my
| recollection).
| tambourine_man wrote:
| Yes, 5 years seems crazily optimistic to me.
|
| But this is Jim Keller and I'm a random internet guy.
| Varloom wrote:
| This man knows exactly what he is talking about. He was
| responsible for designing the original AMD Athlon 64. He
| worked at apple to transition from Generic Samsung ARM SoC to
| their own Apple silicon which is the base for modern M1 Apple
| silicon. He worked for Intel (we'll see his work in Lunar
| Lake, Jim Keller's Royal Core Project). And most importantly
| he worked again at AMD and gave us Zen architecture.
| throwaway4good wrote:
| I find a completely silly claim but I guess as a CEO you need
| to talk your book.
|
| Here is another interview with Jim Keller where he explains
| why instruction sets doesn't matter that much:
|
| https://www.anandtech.com/show/16762/an-anandtech-
| interview-...
| ladberg wrote:
| That's not exactly my takeaway, e.g. he says this in that
| interview which is pretty consistent:
|
| > So if I was just going to say if I want to build a
| computer really fast today, and I want it to go fast,
| RISC-V is the easiest one to choose. It's the simplest one,
| it has got all the right features, it has got the right top
| eight instructions that you actually need to optimize for,
| and it doesn't have too much junk.
| pier25 wrote:
| Who knows. There's a huge financial motivation to move to
| RISC-V to save on energy and heat. Cooling is one of the
| biggest expenses of a data center.
| blueblob wrote:
| Is RISC-V any more efficient than arm?
| pier25 wrote:
| Probably similar but the advantage of RISC-V is that it's
| open source.
|
| AFAIK x86 is still dominating the data centers. There's a
| bit of ARM going on but wouldn't it make more sense to
| switch straight to RISC-V?
| blueblob wrote:
| Yeah, an open source architecture sounds like it could be
| a game changer for the SBC market
| Osiris wrote:
| Considering modern processors spend 4-5 years in development
| before public release, someone would have to be building the
| game changing RISC-V CPU right now.
|
| Maybe he meant that development on RISC-V CPUs would start in
| earnest in the next 5-10 years?
| [deleted]
| snvzz wrote:
| >Considering modern processors spend 4-5 years in development
| before public release, someone would have to be building the
| game changing RISC-V CPU right now.
|
| And they are.
|
| Tenstorrent is working on Ascalon. Wei-han Lien (lead
| architect of M1 at Apple) is the lead architect. Ascalon is a
| RISC-V microarchitecture expected to be released in 2024,
| with similar performance to projected AMD Zen5 (also 2024),
| but lower power consumption.
|
| Ventana Veyron is due late 2023. A very high performance
| server chip AIUI implementing RVA22+V.
|
| Rivos has been working on something RISC-V, with a very
| strong team, for several years now.
|
| SiFive's next iteration of high performance CPUs is expected
| to be strong.
|
| Alibaba group has something in the works, too.
|
| And this is all just the tip of the iceberg. There's way more
| known projects ongoing, and even more that we do not know of.
| syntheweave wrote:
| Data centers are one of the best demographics for adopting new
| architectures because more of the software can be custom-built
| towards a narrow application: Get a Linux stack to build, add
| some network I/O, add some virtualization, and you can do all
| sorts of things.
|
| Client apps have a much harder time making that jump because
| the environment is more holistic, the hardware more varied, and
| the need for "must-have" proprietary apps more imperative.
| imtringued wrote:
| All the Chinese datacenters.
| SirMaster wrote:
| I wonder if it's not so much that Jim is so super smart that he
| can predict the future, or rather that he has such a huge network
| of peers in which he talks to and from that access to industry
| insider information can draw more accurate conclusions about the
| overall direction of the industry than most.
| jlsfiew38 wrote:
| Listening to Jim Keller talk about engineering makes me feel the
| same way I used to feel when I listened to Jeff Beck playing
| guitar. I didn't know if I was inspired or if I should quit.
| euiq wrote:
| After listening to the interview, I totally get what you mean--
| what a wonderful perspective on computers he has.
| cmrdporcupine wrote:
| So calm and clear in exposition of complicated concepts.
| yalogin wrote:
| Never heard of this company till now. Wonder what other notable
| startups are out there. How does one keep tabs on startups?
| zucker42 wrote:
| I first heard of TensTorrent from Dr. Ian Cutress's YT channel
| TechTechPotato. https://youtu.be/AFVDZeg4RVY
|
| It's a good source for trends in hardware.
| re-thc wrote:
| Anandtech has now lost its touch without Ian and some of the
| older staff. Such a piece of history now...
| hiddencost wrote:
| You're here. :P.
| KeyBoardG wrote:
| Sally Foxton-Ward (EETimes) and Ian Cuttress (TechTechPotato)
| started a podcast (only 2 episodes so far I think) listing
| through companies and what they do. I agree it is very hard to
| follow with companies coming and going.
|
| Not sure if linking is allowed:
| https://www.youtube.com/watch?v=pQzf_1AkNXw
| tikkun wrote:
| I have a list of some I'm keeping tabs on, I'll share it here.
| I do it out of a combination of personal obsession plus because
| I want to source angel investments.
|
| Note though that I'm very biased toward AI companies...
|
| Most established, clear product-market fit:
|
| - OpenAI
|
| - Midjourney
|
| - Character ai
|
| - Runway ML
|
| Ones that are interesting:
|
| - Adept AI
|
| - Modal, Banana.dev
|
| - new.computer
|
| - Magic.dev
|
| - Modular (Mojo)
|
| - tiny corp
|
| - Galileo
|
| - Hippo ML
|
| - Tenstorrent
|
| - contextual.ai
|
| - Chroma
|
| - e2b.dev
|
| - Steamship
|
| - Patterns.app
|
| - GGML
|
| Ones that I want to learn more about before deciding:
|
| - Inflection AI
|
| - GetLindy
|
| - Embra
|
| - Jam.dev
|
| - Vocode.dev
|
| That's about 50% of my list. Happy to clean up the rest and
| write a post if there's interest
| PartiallyTyped wrote:
| What about Anthropic?
| rbzz wrote:
| Anthropic feels super underrated. From my experience,
| Claude+ is on par with GPT-4, 100k context model is
| amazing, but because they don't have a product exposed to
| the public, they don't have to burn billions of $ on things
| like ChatGPT.
|
| Also there's a chance they might avoid some cases of direct
| regulation and local bans since they are under media radar.
| CoastalCoder wrote:
| How about Untether?
|
| They're developing a near-memory inference accelerator that
| looks pretty interesting.
| CoastalCoder wrote:
| I'd also be interested to see the list of companies that
| you've written off.
| albertzeyer wrote:
| Add Nnaisense (https://nnaisense.com/) maybe to the list.
| [deleted]
| WanderPanda wrote:
| I'm not sure about them, they seem more like a research +
| consulting shop than working on products
| fork-bomber wrote:
| SiFive
| jabl wrote:
| Esperanto is doing (unless they have folded/pivoted since I
| last checked) RISC-V HW with an AI focus, heavily using the
| RISC-V vector extension.
| mandeepj wrote:
| > How does one keep tabs on startups?
|
| Keep an eye on notable people! They are the one driving the
| trends, industry direction, and narratives.
|
| You can also create Google Alerts. I'm sure there are other
| alternates as well. https://www.google.com/alerts
|
| And, in general - read news!
| potatoboiler wrote:
| I have been wondering the same thing; scouring Wellfound seems
| to yield little of interest, for example.
| elihu wrote:
| > "His history includes stints at Apple, Tesla and AMD."
|
| Kind of a minor point, but it's weird they omitted Intel.
| andy_ppp wrote:
| Here's the correct link, EE Times website went crazy for me.
| https://youtu.be/_MrGNlXRi9M
| artisanspam wrote:
| [flagged]
| WoodenChair wrote:
| > Don't forget his stint at Intel. I met him there and aside
| from his brilliance, the other thing that stuck out to me is
| how short he is.
|
| I downvoted you because I have no idea how commenting on his
| height is significant to this discussion. In fact calling out a
| physical characteristic that is irrelevant is generally to
| make-fun. I'm not saying that's what you were doing here, but
| it's just totally irrelevant.
| artisanspam wrote:
| It wasn't an attempt to make fun of but I can see how you and
| others would think so. I simply thought it was an interesting
| characteristic of him because there is a trend for CEOs to be
| taller than average.
| euiq wrote:
| He doesn't even seem to be remarkably short in the first
| place. Here's a random picture that I found on the internet:
| <https://www.hwlegend.tech/wp-content/uploads/2020/06/Jim-
| Kel...>
| evilturnip wrote:
| lol I'm always fascinated by the weird psychology motivating
| these comments. Must be a troll.
| akkartik wrote:
| Does "IP" in this article mean "intellectual property"?
| gaudat wrote:
| Yes, "source code for chips / custom hardware" as a product is
| mostly referred as IPs. This includes HDL/RTL, netlists and GDS
| layout for a specific process, in increasing order of closeness
| to physical reality.
|
| Tangentially related, it is interesting to note that layouts
| and masks for manufacturing custom chips are not really
| copyrightable, so there is a thing named "mask rights" and (M)
| instead of (C) for them.
| IshKebab wrote:
| Yeah they're talking about hardware designs. Chip designers can
| buy "IP" from vendors for stuff they don't want to implement
| themselves, e.g. PCIe interfaces, embedded CPUs, etc. It's
| delivered as SystemVerilog code, probably obfuscated and maybe
| encrypted (yeah the encryption is theoretically worthless but I
| guess it is a sort of honesty / deniability thing).
| ramshanker wrote:
| Still awaiting the retail release of that 75w PCIe card with out-
| of-box PyTorch support. If it can run, say StableDiffusion /
| LAAMA and family, many more use cases shall emerge.
|
| Obviously retail price will have to be competitive with Gaming
| Cards.
|
| My 2 cents.
| VHRanger wrote:
| We've been waiting on it for 2 years....
|
| When I talked with TensTorrent people they claimed it was a
| $1000 A100 competitor with 16gb RAM. Still waiting on it.
| ramshanker wrote:
| I watched the video, apparently they are targeting $1000-$2000,
| and said even $500 sounds interesting.
|
| Maybe, just maybe a highly cut down version, 15 watt model
| could bring it down to RaspberryPi level price. $45. UsbC
| interface, if not x4 PCIe. Like, people buy it first, and
| decide what to do with it latter on. Guess I am going to far
| down the fantasy lane. ;)
| adgjlsfhk1 wrote:
| Almost certainly not. Raspberry pi economics only work since
| they used a cheap of the shelf CPU. For the first few gens, I
| highly doubt anything below $200 will make any sense given
| the unit profit margins they will need.
| convivialdingo wrote:
| We're having great luck running Yolo 8 models on cheap intel
| n5105 boxes. 8gb ram, 256gb SSD, USB3 for ~$150... Ubuntu
| running ONNX gets us 600ms per frame. Not great, but good
| enough for now.
| FL33TW00D wrote:
| I could most likely make it faster - depending on the YOLO
| used.
___________________________________________________________________
(page generated 2023-06-13 23:02 UTC)