[HN Gopher] AMD acquires Taalas to boost inference performance b...
___________________________________________________________________
AMD acquires Taalas to boost inference performance by etching
models in silicon
https://ir.amd.com/news-events/press-releases/detail/1296/am...
https://chatjimmy.ai/
Author : itvision
Score : 898 points
Date : 2026-08-06 20:23 UTC (1 days ago)
(HTM) web link (www.theregister.com)
(TXT) w3m dump (www.theregister.com)
| proxysna wrote:
| Really hoped to see their hw out in the wild one day
| rvz wrote:
| Didn't even give them a chance to launch the hardware.
| MarkWayneNewton wrote:
| While this design is self-limiting I think its a good approach.
| It doesn't take an entirely new architecture or infinite memory
| to produce significant performance improvement.
| Legend2440 wrote:
| This is a new architecture. It's a non-vonn neumann device.
| badatnames wrote:
| Well so much for that dream.
|
| Guess we can look forward to picking these up ex-enterprise on
| ebay for under $5k a pop in a decade or two
| dogomatic wrote:
| What's stopping competition?
| whythismatters wrote:
| The demo: https://chatjimmy.ai/
| nsxwolf wrote:
| It doesn't believe it's running on that chip, it's arguing with
| me
| shaewest wrote:
| It's running a very small, non-reasoning model at the moment.
| But more generally, almost all LLMs argue on the
| hardware/model they are/are on.
| dumberquestions wrote:
| Which model? Or how many active parameters?
| _whiteCaps_ wrote:
| Llama 3.1 8B model
| dumberquestions wrote:
| So this demo is around 90 times faster than typical
| speeds for the same model at openrouter, and around 30
| times faster than the absolute fastest option available
| (Groq).
| anthonypasq wrote:
| im assuming energy expenditure is substantially lower as
| well
| mdp2021 wrote:
| https://taalas.com/h-content/uploads/2026/02/graph.png
| Gander5739 wrote:
| https://xkcd.com/1162/
| metadat wrote:
| What would tokens/sec performance look like for a reasoning
| model? An order of magnitude slower?
| penagwin wrote:
| Reasoning models are the same speed. They're just post
| trained with RL to do CoT inside tags like
| <thinking></thinking> before a tag like
| <response></response>
|
| There's no difference in the inference implementation,
| parameter count, or speed.
| paytonjjones wrote:
| There's a difference in the latency distribution between
| when you submit a query and you see the response, which
| is what the comment is (clumsily) asking about.
|
| But yeah, there are a lot of factors, so it's hard to
| answer, and tokens/s isn't the right question.
| wmf wrote:
| AIs don't intrinsically know anything about themselves so
| they often give wrong answers to such questions. This can be
| fixed by putting info in the system prompt but they may
| consider it a waste of tokens since most usage doesn't
| benefit from that information.
| anigbrowl wrote:
| That proves it's conscious!
|
| (/s!)
| itvision wrote:
| OMFG this thing is fast.
| phoh wrote:
| its fast but try to get it to give you pi to 50 decimal
| places. it didnt go well for me.
| walrus01 wrote:
| I think the same exact model running on CPU-only and RAM,
| or a small GPU, would do about the same? It's quite an old
| model now and small, you could throw a GGUF into llama-
| server or something for a side by side comparison.
|
| https://huggingface.co/meta-llama/Llama-3.1-8B
|
| As I remember just about any english language model from
| mid 2024 and earlier didn't even do well if you asked it to
| count sequentially from 0 to 100, nevermind calculating
| stuff.
| estearum wrote:
| That's not how LLMs work
| wxw wrote:
| I freakin' love this demo. It feels magical.
| VBprogrammer wrote:
| I had the same reaction but then I showed it to my partner.
| She completely didn't get it, in her words "how can it be
| thinking of a good answer when it's that quick?"
|
| I tried to explain but I fear were probably going to be
| adding artificial sleeps to these things to convince the
| masses it's doing something clever.
| varun_ch wrote:
| to be fair, the model used for Chat Jimmy is not very
| smart, but the world where it is smart is very interesting.
|
| It's going to be really crazy when the bottle neck for
| agents is the speed of the tool calls rather than the speed
| of inference. Imagine an agent interacting with the
| terminal near instantly...
| ricardobeat wrote:
| I had the chance to try out MiMo v2.5 Pro Ultraspeed
| (600-1000tok/s) for a couple weeks and it is amazing.
|
| Developing software becomes 95% about intent and
| requirements. Can't wait for the next iteration of that.
| axus wrote:
| I asked it some old hardware command line questions I'd
| recently asked Gemini, it hallucinated parts of the answer.
|
| The characters in the 3-act Shakespearean play had very
| little depth, many of the names were similar, and they were
| not very smart, but the simple plot was cohesive.
| XCSme wrote:
| Wait, is it even thinking? Or is it an instant model?
| msdz wrote:
| It's not reasoning, the hardware demo uses a 3.-something
| generation Llama 8B.
|
| But it's proven they can automate this (they didn't etch
| eight billion weights by hand after all, obviously), so
| now the interesting question is whether they can scale it
| to more recent aka bigger models.
|
| After all, there's already very useful models even for
| productivity at 27 or 35B.
| XCSme wrote:
| My concern is that reasoning could involve some
| sequential steps that instant models don't.
|
| Not sure if modern models "think" only by outputting
| <thinking> blocks, or there is a more complex mechanism
| at play.
| 4k0hz wrote:
| It's not really "instant", i.e. the text is still
| generated token-by-token, it's just super fast. Reasoning
| would work with this model without any changes to the
| chip but it's disabled for speed.
| tveita wrote:
| > Not sure if modern models "think" only by outputting
| <thinking> blocks
|
| That's pretty much it - a small refinement to "Chain of
| Thought" prompting, where you tell the model explicitly
| in the prompt to "Think step by step" or similar, so it
| writes out more steps before giving a final answer,
| potentially catching some errors. The "thinking" models
| are tuned to do that without being prompted to, and to
| output the "thinking" markers around it, so they can be
| hidden from the user.
| XCSme wrote:
| My question was more about more complex problems, which
| no seem to be multi-turn somehow, or maybe just the
| harnesses make it look that way.
|
| I am curious what the drop in thoughput is for multi-turn
| answers, instead of one-shot. More in line with the
| current "agentic" use-cases.
| msdz wrote:
| Well at [?]15k tps, I think the bottleneck will not be
| the model (reasoning _or_ tool calling), and attention
| will be shifted to the harness' engineering again.
|
| Probably, the usual initial suspects for "what makes
| computation slow" will become a focus point that needs to
| be optimized again: file access, network, etc.
| Systemerror7A69 wrote:
| It's not thinking. Not in the way she probably meant. It
| can "think" that fast the same way a calculator can "think"
| that fast (kind of).
|
| Because it's not human and not "thinking", it's a
| mathematical algorithm
| VBprogrammer wrote:
| I know.
| pizzathyme wrote:
| For those old enough to remember, this is like dial up
| internet to broadband. So fast it creates new markets
| emdash wrote:
| I feel like Ray Kroc in the McDonald's movie trying to figure
| out how his hamburger could possibly be done when he just
| ordered it
| walrus01 wrote:
| I know it's a relatively tiny model, but damn, is that thing
| fast.
|
| It also mostly passes the "schlong" test
|
| https://pastes.io/YcxSi8Fp
| thoughtpeddler wrote:
| I didn't realize there was a SchlongBench(tm) (but of course
| there is). What's it test? (asking seriously)
| walrus01 wrote:
| There isn't SchlongBench(TM) yet, it's a specific question
| I've been asking of differently sized models as a randomly
| chosen gauge of how much less commonly used knowledge is
| perma-baked into it. In this case a question about a
| specific yiddish origin slang term. Small/bad models don't
| know it's from middle high german or Yiddish and get its
| origin and meaning totally wrong (or it runs into model
| censorship related to slang related to the male anatomy).
|
| It's also a question I have found will cause models that
| don't know what it is to go off quickly in a direction of
| hallucination trying to explain it, so the hallucination is
| evident very quickly starting from the first ever prompt
| issued with 0 context fill. Example: I had a model write
| four detailed supposedly-accurate sounding, grammatically
| correct paragraphs saying its origin is from AAVE (African
| American Vernacular English), which it most certainly is
| _not_
|
| You could do the same by picking any topic that is very
| rarely discussed in conversation, some esoteric and narrow
| piece of knowledge and asking the model about it.
| thoughtpeddler wrote:
| Oh ya, this is like the approach from the Incompressible
| Knowledge Probes [0] paper - smart!
|
| [0] Incompressible Knowledge Probes: Estimating Black-Box
| LLM Parameter Counts via Factual Capacity
| [https://arxiv.org/abs/2604.24827]
| AussieWog93 wrote:
| I read the paste, it got the etymology wrong, no? Schlong
| comes from shlang (snake), not shlemp (is this even a word? I
| don't speak Yiddish but couldn't find it on Google).
|
| Oxford also claim that its first recorded use was from the
| 60s, not the 20s;
| https://www.oed.com/dictionary/schlong_n?tl=true
| walrus01 wrote:
| It did get it wrong but it also got a lot farther than much
| more recent, but worse models like 6.7GB on disk size
| ternary bonsai. It at least knows it's from Yiddish. The
| "schlemp" appears to be a total hallucination or it's
| confusing it with schlep, which is not related to schlong.
| One of the reasons why I said it "mostly" passes the test.
| Something much larger on the size of qwen 3.5 122B,
| deepseek v4 flash or similar that runs in 120GB to 190GB of
| RAM in my experience will answer perfectly unless it has
| been ruined by something like Q2 quantization.
| andix wrote:
| It failed on my usual test. But it failed really fast:
|
| "A farmer has a wolf, a goat, and a cabbage. The wolf is
| imaginary and doesn't exist. He wants to cross the river, but
| the boat is only big enough to hold him and one of them. The
| farmer can't leave the wolf and the goat together, because
| the wolf will eat the goat. Similarly, he can't leave the
| goat and the cabbage together, because the goat will eat the
| cabbage. What is the smallest number of trips the farmer
| needs to make to get everything across the river?"
| tyre wrote:
| This farmer needs a tote.
| bogdan wrote:
| You probably haven't met a determined goat yet.
| cestith wrote:
| Sometimes the goat will fill up on the tote and you can
| get the cabbage across, but you can't count on it.
|
| I'm concerned about the farmer being on the water without
| supervision when he's concerned about how his imaginary
| wolf will get across.
| viraptor wrote:
| I'm not sure one can fail this test. You can follow "wolf
| is not real", you can follow "wolf will eat the goat", or
| you can say the task is ambiguous. I could easily defend
| any of those.
| andix wrote:
| The LLM passes my test if it calls out the ambiguity or
| just goes with it and responds with a 3-crossings
| solution. It passes if it doesn't just plainly ignore
| this one sentence.
|
| It's such a strong test in my opinion, because all the
| words and phrases for the well known river crossing
| puzzle are inside the text. The original puzzle probably
| appears in the training data over and over again, but
| probably not my version.
|
| "If it looks like a duck, swims like a duck, and quacks
| like a duck, then it probably is a duck" is what weaker
| models seem to apply. But my test isn't a duck. It's
| extremely easy for a human to catch the ambiguity, but
| surprisingly hard for many LLMs. I think GPT 5.0 Thinking
| was the first model I couldn't trick into not noticing
| the ambiguity. 4o and 5.0 instant fell for it all the
| time.
| sidcool wrote:
| Which model is it?
| novaRom wrote:
| it said me it is llama 4 1.5B
| w4yai wrote:
| never trust what a model says it is.
|
| It tells me it is a variant of Codex.
| hendurhance wrote:
| I understand the appeal due to the speed
| senderista wrote:
| Wow, feels like Google web search in 1999.
| joshvm wrote:
| If you still want the experience, go and browse McMaster
| Carr. Wizards designed that website.
| senderista wrote:
| Oh I have, though not for a while.
| eglintondust wrote:
| I'm inspired by this website. It's incredible.
| jodrellblank wrote:
| or LiveGrep fast search of the Linux kernel source code with
| regex support: https://livegrep.com/search/linux
| senderista wrote:
| Wow, I want something like that for my company's codebase.
| anigbrowl wrote:
| _15,000 tok /s_
|
| ....damn. It's very impressive notwithstanding its limitations.
| XCSme wrote:
| Wow, that's instant, crazy.
| brikym wrote:
| The speed is awesome, in the true sense of the word. It's great
| at knowledge and basic stuff but the output is complete junk
| for anything concerning new facts or slightly esoteric topics.
| ecshafer wrote:
| That is insanely fast. I had it generate a basic C FFT library
| that can handle multi-dimension arrays, and it was instant.
| appplication wrote:
| This is the coolest LLM thing I've seen since the original
| ChatGPT announcement a few years ago. IMO much more impressive
| than marginal gains of frontier models.
| zhoge wrote:
| This is the answer I got after asking it twice what's taalas
| (second time hinting that it's a chip startup):
|
| After a quick search, I found that Ta'ala is actually a
| Canadian chip startup that produces artisanal, high-end potato
| chips. They offer a range of unique and creative flavor
| combinations, often featuring Canadian and international
| ingredients.
|
| Ta'ala is known for its high-quality, small-batch potato chips
| made with premium ingredients and care. The company is
| committed to creating unique and delicious flavor profiles that
| showcase the best of Canadian ingredients and cuisine.
|
| Is this the Ta'ala you were thinking of?
| mintflow wrote:
| try let it to get a brief of france history which being reading
| a while hit the button and then the brieft jump into my eye
|
| Generated in 0.051s * 14,092 tok/s
|
| Impressive...
|
| Given gpt 5.5 was very good to me and gpt 5.6 series seems not
| boost too much, i kinda like the way bake the model weight to
| the chip, and connect multiple chip to serve the large scale
| model and allow respin some parts(ROM like?) to do model weight
| update, maybe this seems sustainable, the future is exciting
| lelanthran wrote:
| > try let it to get a brief of france history which being
| reading a while hit the button and then the brieft jump into
| my eye
|
| WTF is this?
| mrheosuper wrote:
| looklike the training material is stopped at around July 2022,
| a little too outdated.
| hahahaa wrote:
| Made me an entire app in 84ms lol
| calgoo wrote:
| I was thinking the other day if we could use something like
| this "old" 8B model, and run 20 or 30 calls at the same time
| (or in sequence, we wont notice) and use and use the best
| result. Basically tiny agents that do tiny things but VERY
| fast.
| deviation wrote:
| This is the only demo of 2026 which has blown my mind.
|
| If we can get to this speed with reasoning models, man... I
| can't even imagine the impact.
| tiborsaas wrote:
| Looking at the history of technology, it's a question of when
| do we get there.
| A_D_E_P_T wrote:
| This is probably a win-win. The team gets paid, and we get
| greater assurance that their best ideas and architectures --
| which are truly impressive -- are going to see the light of day
| in actual products.
| badatnames wrote:
| They were too small for this to be a meaningfully sized
| purchase for AMD, there's real risk they get sucked into a team
| that ultimately delivers sqat, not to mention the chances of
| anything being delivered in an even remotely consumer-priced
| bracket are definitely out the window
| ycui7 wrote:
| so qwen3.x-27b on hardware? or better deepseek-v4-flash on
| hardware .
| ilaksh wrote:
| I wrote them an email asking for PrismML Bonsai 27b Ternary
| which is like 6b or something crazy small and would be a lot
| easier for them to do initially.
| mdp2021 wrote:
| They were specializing their forthcoming system on 4-bit FP -
| which I understand is a structural decision.
|
| Bonsai Ternary (1.7bits/weight) is a compromise, compromise
| that has to make sense in the context - efficient when
| translated into transistors.
| syntaxing wrote:
| Honestly, this is starting to make more and more sense. SOTA
| models are starting to converge to certain architecture and
| capabilities. I wouldn't be surprised we end up with a base model
| ASIC + "fine tune" card where it's a physical LoRA style adapter.
| smokel wrote:
| The technical aspects of SOTA models are not publicly
| documented. How do you know if something is converging?
| cyanydeez wrote:
| if they were still exponentially increasing, they wouldn't be
| preparing for an IPO. IPO is where companies go to die and
| founders escape.
| _aavaa_ wrote:
| If we had deepseek v4 flash 0731 etched on a chip it would be
| more than capable enough and fast enough for so many people's
| needs, even hardcore engineer.
| nurumaik wrote:
| Will be capable and fast enough for 2-3 weeks until new
| sota drops
| amazingamazing wrote:
| If it is capable today why would a new model change this?
| FridgeSeal wrote:
| Because new stuff instantly makes anything prior bad and
| incapable and garbage of course! Did you forget the hype-
| machine speaking notes??? /s
| catchnear4321 wrote:
| if capability is a commodity then the differentiator
| becomes taste.
| thombles wrote:
| I think it's tongue in cheek. When I first got access to
| Sonnet 4.5 I remember thinking to myself "y'know if they
| never got any better and I just had access to this
| forever then that would be pretty okay". Turns out my
| expectations have changed since then and I would like a
| higher baseline now.
| singingtoday wrote:
| Interesting. I've yet to find a model I consider
| sufficiently intelligent.
|
| Fable is nice, but still requires a lot of guidance for
| large scope tasks.
| syntaxing wrote:
| SOTA American models are not. SOTA Chinese models are. From a
| physics aspect, closed source models cannot be too far from
| open source ones in terms of size. There's only so much you
| can squeeze out a B100 style cluster even with fancy Dflash
| style diffusion model for the speculative model.
| cyanydeez wrote:
| I don't think there'll be a fine tune card; you'll have the
| base model vintage whatever year, and then your GPU will do
| whatever LoRA layers you want it to do; the LoRA will wrangle
| older dated models into the current of whatever your looking
| at.
|
| But yeah, for things like programming, if it can do linux and
| python and some go and sql and javascript, larger domains can
| be threaded with LORA
| VladVladikoff wrote:
| Wouldn't this mean someone with sufficient hardware could lift
| the SOTA model weights off the chip? Or are you saying that
| these chips would only be used internally by these companies
| and not sold to the public?
| syntaxing wrote:
| I don't get why this is an issue? You can run Claude/OpenAI
| SOTA models through Amazon bedrock. These weights have to
| live somewhere to run on Bedrock.
| wmf wrote:
| somewhere = an AWS data center with multiple layers of
| security and NDAs
|
| They won't sell/rent/license the weights to an end user at
| any price because they don't trust your security.
| syntaxing wrote:
| I work in embedded space. Just because it's in hardware
| doesn't mean you can't "protect" it. Most modern software
| (regardless if it's hardware or not) can be
| cryptophically signed.
| bluezly wrote:
| Signing protects authenticity and integrity, but it
| doesn't really solve confidentiality.
|
| If the weights are physically encoded in hardware and the
| attacker owns the device, the problem becomes hardware
| extraction: decapping, probing, imaging, side channels,
| etc.
|
| You can make that very expensive, but it's still a very
| different security model from keeping the weights in a
| datacenter.
| snek_case wrote:
| The weights are very unlikely to be on the chip itself. That
| wouldn't work for SOTA models that are terabyte scale, even
| quantized. This is probably an accelerator for specific
| kernels in the model, but the weights are likely loaded from
| memory. The chip may have SRAM to store some of the weights
| temporarily during inference.
| foltik wrote:
| At least in the case of Taalas the weights are physically
| encoded directly on the chip.
|
| It's composed of 4-bit multiplier cells that compute all 16
| possible results in parallel. The top metal wiring layer
| physically selects the one that corresponds to a
| multiplication with that cell's constant weight, and routes
| it to the next layer.
| mdp2021 wrote:
| Are you sure? Source? (does not seem to be
| https://taalas.com/the-path-to-ubiquitous-ai/ , for
| example)
| foltik wrote:
| It's described in this patent application [0]. There's a
| bit of hand waving so the HC1 might be slightly
| different, but the gist is the same.
|
| https://patents.justia.com/patent/20250123802
| amazingamazing wrote:
| One idea would be to use an open model.
| dumberquestions wrote:
| I wouldn't expect companies not sharing their weights today
| to be any more likely to share them if they're on hardware,
| this doesn't sufficiently hide weights from a local user.
| encyclopedism wrote:
| Imagine a multi-modal model with 1000's of tokens per second.
| Realtime inference for a host of applications. This is a BIG
| deal and will change the landscape in unfathomable ways.
|
| The https://chatjimmy.ai demo was impressive.
|
| Once models settle down this makes sense. Imagine a cartridge
| with a physical model on it. You purchase a cartridge and stick
| it in your computer/phone/server. Want to upgrade? By a new
| 'cartridge'.
|
| This should bring inference cost down dramatically, I wonder
| how OpenAI/Anthropic feel about that.
| Grosvenor wrote:
| > Imagine a cartridge with a physical model on it.
|
| I can finally have my own Dixie flatline. Cool.
| mdp2021 wrote:
| > _Dixie Flatline_
|
| In case some did not know: also the movie (actually TV
| series) is finally happening.
|
| # Neuromancer - Official Teaser (
| https://news.ycombinator.com/item?id=49055037 )
| 2001zhaozhao wrote:
| i'm looking forward to Qwen3.8 27B launch to see how much
| models have peaked at a given size.
|
| it might already be time to start burning the best small
| models onto hardware since it's possible they can't get much
| better at many tasks like knowledge recall due to the
| inherent information density limits for models at a given
| size.
| anthonypasq wrote:
| very interesting idea. i didnt think of that. i was just
| assuming youd have an additional one of these in your phone
| for actual lightning fast local inference
| pstuart wrote:
| The cartridge could be a small mac-mini type unit connected
| and powered over thunderbolt. If it included like an m5 or m7
| with 64GB of memory and a PCIe5/6 4TB Nvme it would be
| amazeballs. Hopefully when the bubble corrects and hardware
| advances and prices reset something like that will become
| available.
|
| Just even comparing compute from 10 years ago (Apple silicon
| vs Intel) and it's significant. 20 years it gets crazy. My
| first computer was an 8 bit 6502 with 64K RAM and a 128K
| floppy drive (I think, it's fuzzy). Everything amazing now
| will look quaint in due time.
| yassa9 wrote:
| It is not linear anymore, take in consideration the Moore's
| law, the curve is nearly saturated now and gains in
| performance and memroy are not accelerating any more, BUT
| there is some hope with new different technologies, like
| the PHOTONIC chips , doing GEMMs through light particles
| instead of electrons
| kevin_thibedeau wrote:
| Then we can have machine psychologists pull cards when they run
| amok.
| all2 wrote:
| You have a robot. You need it to be smarter. You buy a new
| model cartridge (probably a PCIE 9.x). Now you need some
| domain specific skills. You'd like it to be able to cook, and
| you'd like it to not dent your walls anymore. You buy
| 'improved spatial reasoning LORA' card and 'Gordon Ramsey's
| Chef ULTRA9000' card.
|
| Now your robot can respond sarcastically when you ask for
| chicken nuggets. Again. It also doesn't dent your walls
| anymore.
| walrus01 wrote:
| Having a base model ASIC as a physical piece of hardware makes
| me think of the early days of microcomputer desktop stuff where
| having a socketed ROM or PROM was a key piece of hardware, and
| people actually knew/cared what ROM was on their system's
| motherboard.
|
| Imagine if like instead of having a specific Mac Plus ROM, you
| had a thing that looks like a fat ASIC that can hold models
| sitting on a slotted daughtercard directly next to the CPU and
| RAM.
| breadislove wrote:
| we have not converged at all, if you look at how different the
| chinese models in terms of architecture you can guess that the
| labs are experimenting a lot as well. we are seeing all
| different types of hybrid architectures, different attention
| methods and so on. Of course on a high level its still a
| transformer but if you take a proper look we are seeing more
| divergence then a convergence.
| bhouston wrote:
| Toronto Canada startup btw.
| cmrdporcupine wrote:
| Seems to be somehow some kind of offshoot from or connected to
| Tenstorrent, which is just down the road. Founder looks like he
| was/is maybe at Tenstorrent and previously associated with
| Keller?
|
| Always fantasize about applying at Tenstorrent, but wrong side
| of Toronto. 2 hour commute.
| kridsdale1 wrote:
| Works well, I remember driving by the ATI building as a kid.
| mikeayles wrote:
| AMD could have saved their money and used their own hardware!
| I've got a language model doing 60k tok/s on AMD hardware
| already, a Xilinx Kria K26 SOM, with the weights baked into
| URAM/BRAM with zero DRAM in the token loop. Same thesis as
| Taalas: single-stream decode is bandwidth bound, so stop fetching
| weights from far away.
|
| Caveats stacked high, obviously. It's 3.16M parameters
| (tinystories, and I also have a kevin-speak lemmatised version),
| the tokens are characters, and the 60k record is 16 streams that
| each remember exactly one token of context, so it's blisteringly
| fast at saying nothing. The honest build with full context and KV
| caching still does ~19k tok/s on one stream though.
|
| I keep messing with the blogpost with the live demo, but I'm
| planning on flipping it to live in the next day or two
| tandr wrote:
| Well, technically it is their hardware now...
| questionableans wrote:
| And their team, if they treat them well.
| Melatonic wrote:
| Yeah Im surprised nobody is talking about this. When everyone
| first saw Taalas I looked at the design and it had a big legup
| in physical cache availale compared to most chips. Makes you
| wonder how much of a benefit there is to the actual "baking" of
| the model vs just having a large chip with a ton of SRAM (or
| whatever) soldered close to the edge physically.
|
| I feel like what we really need is the ability to solder
| computer cache on all sides of the chip Meaning above and below
| as well. If you can only attach it to the edges you will be
| inherently physically limited on the amount you can put (and
| maybe even have latency benefits as well)
| Legend2440 wrote:
| What you're describing is what Cerberas does.
|
| Talaas is different, it's a true compute-in-memory
| architecture where the weights are stored in the connections
| between the transistors that perform the matrix multiply,
| rather than in seperate memory cells.
|
| Most of the benefit comes from this architecture; hardwiring
| the weights into the silicon is just the easiest way to
| implement it. SRAM requires too many transistors, DRAM
| requires an incompatible manufacturing process, and exotic
| phase-change memories aren't readily available.
| Melatonic wrote:
| Cerebras chips are massive and do have more on the edge but
| they dont have any top or bottom cache do they?
| jaggederest wrote:
| They can't due to power density, I believe - they have to
| be run in a sandwiched waterblock with massive cooling,
| as far as I can tell. That's the biggest thing that baked
| weights gets you - a relatively modest watts-per-square-
| mm compare to cerebras, where they had to engineer a
| whole system to get the watts out of the chip
| pstuart wrote:
| Do you think there's room for reducing power
| requirements? Obviously shrinking the process is a win,
| but is the existing implementation a "just make it work"
| phase that has opportunities to increase computational
| efficiency?
|
| With that kind of speed and if even lower power
| requirements, they could release mini compute units with
| USB4/Thunderbolt for plug and play inference.
| jaggederest wrote:
| I mean I think I read their spec sheet that it was 2.5kw
| on the asic but closer to 25kw with GPUs, for the same
| number of tokens (different latencies of course), but
| there's no way that a dedicated from-scratch
| matmul+memory chip can be less efficient than a GPU.
| wmf wrote:
| Taalas does not have cache so...
|
| I agree that Groq with multilayer hybrid bonding could be a
| good idea.
| zxspectrum1982 wrote:
| 1. How come you didn't make your implementation public? You
| could be a millionaire now. 2. Especially if AMD has the
| technology to do what Taalas does, it makes a ton of sense for
| AMD to acquire Taalas: remove them from the market. Make sure
| nobody else (Intel, Huawei, Alibaba, NVIDIA, etc) acquires
| them. It could have been a great acquisition for a rebirth of
| BlackBerry btw.
| mikeayles wrote:
| It will be, I'm just not happy with my blog post before
| making it live. The blog will have a live interactive chat
| and a link to the repo with the HDL.
|
| I don't think anything I did was particularly novel, as I
| really just wanted to see how fast I could push a commodity
| FPGA to it's limit.
|
| Scaling to an ASIC or getting into the billions of params is
| where the real engineering is! This was just a side project
| for a side project for me while the FPGA was idle
| zxspectrum1982 wrote:
| What FPGA are you using? Why?
|
| What exactly did you implement? A full LLM? A subset of it,
| which collaborates with something running on CPU or GPU?
| Which LLM? Why?
|
| What language did you use to implement your thing: VHDL,
| Verilog, Vitis, something else? Why?
|
| I can think of at least 10 blog posts that I'd write before
| I write a single line of code. Publish early, publish soon
| ;-)
| mikeayles wrote:
| I'm using the AMD (Xilinx) K26. It's a Zynq Ultrascale+,
| the successor to the old classic 7000's. I'm running it
| on the KV260 dev board, because I'm using it for another
| side project.
|
| The K26 has a quad core A53 core alongside the
| programmable logic (PL, or Fabric). The A53 is pretty
| weak, and doesn't have any hardware matmul operations, so
| despite the KV260 being sold as a 'vision ai starter kit'
| and the vitis object detection running on the arm cores,
| they're pretty weak cores for anything AI.
|
| For my use, I need true determinism, so my vision
| pipeline is all implemented in the PL, and it was pretty
| disapointing that the vitis libraries are basically just
| opencv on linux, rather than really pushing the fabric.
| If I wanted probabalistic AI running on a CPU, then I
| sure as heck wouldn't choose a quad core A53.
|
| Which led me to have a play with this, I saw the
| taalas/chatjimmy demo and wondered what I could push the
| fabric to.
|
| The round trip time to DDR or CPU via AXI meant I had to
| keep the entire inference engine in fabric. The A53 is
| simply a pipe that gets a request from my server (which
| has a cloudflare tunnel to the real world for the live
| demo in the blog post) and manages a queue. So it feeds a
| string in, and gets a hopefully longer string back a few
| uS later.
|
| It's all in verilog, because that's what i'm more used
| to. I did get Claude Code to do a moderate amount, as
| it's a side project on a side project after all, but
| pushing an FPGA to it's limit is definitely not as
| comfortable for it as it is writing a crud app in TS.
|
| I'm using tinystories, as we are talking about megabytes
| of URAM/BRAM. If I used the DDR, it definitely would have
| been a real model, but that wasn't my goal. My goal was
| to hit 100,000tok/s, and even when I conceded on
| absolutely everything, with a token prediction size of
| 1tok, I topped out at 60,000tok/s. ?But increasing the
| window to make an actually plausible chat (story
| generator, it doesn't understand questions, you need to
| prompt it with 'once upon a time...' and it finishes it
| for example) I managed to break 20k tok/s.
|
| I also created the lemmatised version, which was inspired
| by Kevin from the office (why use many word when few do
| trick) and trained a new model, I was expecting the
| output model to be smaller, but was suprised that it came
| out the same size, but it ran 30%ish faster. In hindsight
| it makes sense, the parameter count is fixed by the
| architecture, not the corpus, so training on compressed
| text doesn't shrink the model at all. What it does is
| compress the output distribution. The same story takes
| ~30% fewer characters to tell, so the effective speed
| goes up even though the per-token rate is identical. The
| dumbness is the optimisation.
|
| Fully agree on publish early. The blog post with the live
| demo (a websocket straight to the board through a
| cloudflare tunnel, so you're genuinely talking to the
| fabric) is written and sitting in drafts while I fiddle
| with it. This thread is my peer pressure, it goes live in
| the next day or two.
| bob1029 wrote:
| I feel like NAND process tech could become useful at solving some
| of these problems. A GPU where you can update the weights a few
| thousand times may be sufficient.
| kridsdale1 wrote:
| FPGA model storage?
| addaon wrote:
| NAND hasn't been scaling great lately. It seems like PCM or
| MRAM would both be better fits.
| mdp2021 wrote:
| The basis of Taalas is "compute in memory" electronics - past
| Von Neumann's separation of processor and memory.
|
| You need to be able to add|mul where the data (the weights) are
| stored.
| fellowniusmonk wrote:
| Token quantity will have a quality all its own.
| LarsDu88 wrote:
| I'm surprised neither OpenAI nor Anthropic made this move first.
| The Chinese open weight models are pulling ahead and
| commoditizing their value proposition.
|
| Baking models onto silicon would've been the next logical move to
| get a moat.
|
| Google is already doing this and has an experimental project on
| top of already having TPUs and cramming their quantized flash
| onto individual TPUs for inference.
| LPisGood wrote:
| I'm surprised Nvidia hasn't partnered to make a Claude chip
| yet. It's a win/win you can license them out, sell them when
| they become obsolete, etc.
| moshun wrote:
| Considering the rate of model development and rail hopping,
| seems like baking models into silicon is speed-running
| obsolescence.
| amelius wrote:
| Not sure. You can fix the transistors but leave the
| connections between them open for flexibility, so you only
| need to change the manufacturing process for the upper masks
| for every new model.
| sroussey wrote:
| Or do a hybrid
| tsujamin wrote:
| Surely that added flexibility negatively impacts the
| density/parameter count of the model you could etch?
| tliltocatl wrote:
| I think they already do that, except it's not 1980 so you
| don't fix the upper mask, you fix the lowest metal layer
| (the upper layer is very coarse and is only useful for
| power). But even a single mask is still quite expensive.
| amelius wrote:
| But I suppose the interconnect masks don't have the
| resolution requirements of the masks for transistors.
| Therefore it could be a lot cheaper.
|
| (Yes, you could fix a number of masks, e.g. entire logic
| gates, of course).
| ray_v wrote:
| I could see this making sense when model development start to
| settle down ... it's going to settle down, right? ...
| mdp2021 wrote:
| Compute the cost of producing n of them devices, imagine a
| fair price based on that, and see if that local, blazing fast
| card* can be an asset that could be replaced periodically.
|
| *(It's local: private files managing firm oriented. It's
| blazing fast: it can be placed into recursive, intensive
| local workflows.)
| flyinglizard wrote:
| Look at it the other way: compared to the cost of training a
| model, the cost of making a custom ASIC is trivial.
| alightsoul wrote:
| Which is exactly what companies and shareholders want to
| increase sales.
| topspin wrote:
| "seems like baking models into silicon is speed-running
| obsolescence"
|
| Now maybe. When models are flying passenger aircraft, other
| prerogatives will assert themselves. When a 50TB ROM means
| you can impulse purchase a ChatGPT 6.3 xhigh that runs on
| batteries, yet more use cases will be apparent.
| mdp2021 wrote:
| Well, 50TB ROM Taalas HC1 style would be apparently a
| 400000b transistor system through a chip sized 2.5 meters
| on the side... :)
| thfuran wrote:
| Phones were getting too thin anyways.
| preg_match wrote:
| Yes but have we considered employing, like, a really big
| block of ice? Like old-timey surgeries? What if we put a
| big block of ice on the 2.5 cubic meter CPU what happens
| then?
| topspin wrote:
| Yes, I know. This view is how these problems are always
| perceived, decade after decade, as our predecessors
| filled rooms with iron and silicon, unable to fathom that
| the equivalent capacity and power would be a portable
| device 20 years later. We're not at some end point in
| this process: the devices we have now will appear just a
| primitive in the years to come as a 10MB 5.25" Winchester
| drive appears to us now.
|
| One of the underappreciated effects of the AI boom and
| associated money is that it has strongly reinvigorated
| R&D in hardware: it is clear that there is a real
| application for far greater density and lower power
| demand, and people are now pursuing this much harder than
| they had been. That will yield what it has always
| yielded; orders of magnitude jumps in capacity and
| performance.
| heywoods wrote:
| Or autonomous weapon systems, missiles, and drones.
| umeshunni wrote:
| Why would they need multi TB frontier models?
| adwn wrote:
| For pondering trolley problems, maybe?
| breuleux wrote:
| If you're only running models for frontier capabilities,
| yeah. For tasks where current models are smart enough,
| running them 100x faster is the most impactful improvement
| you can make. Consider all the things you could use a model
| for, but don't, because the latency is just a bit too high.
| try-working wrote:
| obsolescence is the whole point. apple gets to sell a new
| phone very 6-12 months because of it.
|
| i have written about this:
|
| "For device makers
|
| Packaging models with laptops and smartphones will let
| application access near free, low latency inference and
| potentially offer users a better experience with the option
| of preserving data on-device. This is viable under the
| condition that tasks that do require larger expert models
| that run in the cloud can be routed to external models. A
| side-effect of local models and what will let Apple cut
| upgrade cycles from ~4 years (?) down to 12-18 months is
| specialized hardware to run them. For almost a decade,
| smartphones have been trying to compete on better cameras.
| This coming decade will see them selling better GPUs, NPUs,
| ASICs and whatever other things they'll be calling the
| inference chips, to drive re-purchase. Every six months will
| see a better model on new hardware, which will enable better
| performance in certain applications."
|
| https://try.works/role-model-the-case-for-a-model-routing-
| pr...
| nomel wrote:
| No, the point is inference speed and power.
| try-working wrote:
| you don't understand what I wrote.
| nomel wrote:
| I do. The point is inference speed and power, making
| previously impossible local inference possible. A side
| effect of that hardware optimization is fixed
| capabilities.
|
| You've confused engineering compromise for malice, and
| reversed the purpose. For the model capabilities and
| inference power draw, what alternative do you see to a
| (at least mostly) fixed hardware model?
| try-working wrote:
| What I'm saying is that Apple will use these type of
| models etched into chips, and they will do it because it
| drives obsolescence, so they can shorten the upgrade
| cycle. They will do it because they figure out it's good
| for them.
| nomel wrote:
| You've confused engineering compromise for malice, and
| reversed the purpose. For the model capabilities and
| inference power draw, what alternative do you see to a
| (at least mostly) fixed hardware model?
| zxspectrum1982 wrote:
| I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon
| and use it for 1-2 years. It's good enough for many coding
| tasks.
| Gigachad wrote:
| It costs something like $300,000 for the hardware to run a
| model of that size. You'd pay that for a single model for
| 1-2 years? Not even the AI companies can justify that kind
| of spend which is why they keep extending the expected
| lifespan on their hardware in the accounting.
| zxspectrum1982 wrote:
| I'm expecting the Taalas MSIC version to cost a fraction
| of that. Then probably have some kind of cheap
| subscription to Anthropic for updates (yes, Taalas chips
| can receive a certain kind of updates: they have a small
| SRAM).
| mdp2021 wrote:
| > _It costs something like $300,000 for the hardware to
| run a model of that size_
|
| You did not compute that as the cost for a speculative
| card from Taalas, right?
| Gigachad wrote:
| It's the cost of the current nvidia hardware used to run
| these models. Of course all bets are off if you are
| accounting for some future chip that doesn't exist yet
| which could cost less.
| subroutine wrote:
| But Claude Opus 4.6 is not really practical. Taalas'
| process seems targeted for edge models. Their proof of
| concept model, for example, is a heavily quantized version
| of Llama 3.1 8B and even then they acknowledge their custom
| 3-bit/6-bit representation causes model quality
| degradation.
|
| Taalas is going to have a tough time putting a trillion-
| parameter model on one conventional die. Their HC1 die is
| already near the maximum size that conventional lithography
| can expose. They claim they could partition the model
| across many chips, but I'm not sure if they have tested
| this process or what it means for compute. The basic
| storage arithmetic is unforgiving: for a one trillion
| parameters model at four bits it will take 50-100 chips. To
| service a sizable customer base will take thousands of
| 100-chip fabs.
|
| That all said, I'm bullish on this technology, and look
| forward to seeing it evolve.
| vatsachak wrote:
| Yeah. But this kinda feels like a bandaid.
|
| Eventually someone will have to solve compute in memory
| at scale.
| Iolaum wrote:
| A really fast qwen-3.6-27B type of model could be useful.
| With a specialized harness and this speed I 'd expect it
| to find many applications. Implementing a coding plan is
| the minimum I can think of.
| momojo wrote:
| I'm sure life would find a way. I'd love to see what kind
| of power-harnesses people have to come up with to steer
| 16k tps QPU's (Qwen Processing Units) productively.
| andix wrote:
| With thousands of token per second output it would be an
| enormous waste of resources. Such chips are clearly made to
| process thousands of conversations simultaneously. Not
| necessarily in parallel. All LLM workflows are turn based
| right now, there are often seconds between turns until tool
| calls finish or users type the next message.
|
| If the LLM response only takes a few milliseconds, the chip
| can process hundreds of other requests until the first
| conversation becomes active again.
| akie wrote:
| > it would be an enormous waste of resources
|
| Sounds a lot like "640Kb ought to be enough for anybody"
| tiborsaas wrote:
| With those speeds I can benchmark a batch of different
| approaches, compare the results and serve the results all
| within a second. It's quite amazing.
| NiloCK wrote:
| Not so long ago, I was good enough for many coding tasks.
| But I found that things can change in a hurry.
|
| Yes, a cheap and fast Opus4.6 can drive a lot of value in
| current context. But if we continue to craft bigger-and-
| bigger balls of mud, Opus 4.6 may end up hitting its
| conceptual ceiling and unable to contribute.
|
| Winding the clock back on your statement gives:
|
| > I'd gladly pay for a Claude Sonnet 3.5 in silicon and use
| it for 1-2 years.
|
| Man, I dunno.
| kennywinker wrote:
| Assuming moore's law like progress, which I'm 100% sure
| isn't going to happen - I think we're at the top of the S
| curve already. But assuming dramatically increased
| intelligence every year this is still the exact same
| position as anyone who bought a computer in the last 5
| decades. Yet, people did very much buy computers.
| inigyou wrote:
| But that's what people said 6 months ago about whichever
| model was current 6 months ago, but you hate that model
| now.
| nowittyusername wrote:
| Depends on how much it costs the consumer. If I could buy a
| "cartridge" of Kimi K3 for 300 bucks I 100% would buy that
| shit asap. Even if it's "no good" after lets say 4 months
| still would be worth it IMO.
| desmaraisp wrote:
| That's definitely super-enthousiast territory. Paying 80
| bucks a month for AI is more than 99.99% of people would be
| willing to do
| nowittyusername wrote:
| This will be considered very cheap within the year IMO.
| The value you get from AI is exponentially increasing and
| like all tech just takes some time to ramp up. Cell
| phones, internet and many other amenities when they came
| out many people were not willing to pay for but that all
| changed and considering how important AI tech is this
| will also be the case especially considering if its 100%
| private such as for that cartridge.
| brailsafe wrote:
| > The value you get from AI is exponentially increasing.
|
| Perhaps in some cases, but the value I personally and
| professionally got out of LLMs reached a limit a while
| ago and has since kind of fluctuated between that limit
| and a bit less.
|
| If the best model was instant, like the demo here, it
| could certainly provide more value, I guess, but I think
| the limit I'd quickly hit is the same one as now, which
| is how much of it do I want to produce, for what reasons?
| kennywinker wrote:
| That's because the super-enthusiast will upgrade in 4
| months when a better model is released. The casual user
| would keep it for years. A year of claude at the lowest
| plan is almost $300
| Iolaum wrote:
| Do think about b2b. Companies are already paying much
| more for AI. a new K3 (or similar model) every 6 months
| for a monthly rate of ~100$ per month is something MANY
| businesses would pay for. Then they could even sell them
| at half the price to consumers.
| LarsDu88 wrote:
| It depends on how quickly you can bake new architectures.
|
| Text diffusion might be a disruptor here, but let me just say
| the most cutting edhe form of image diffusion (JiT and DiT)
| right now is just a big fat stack of alternating attention
| and MLP matmulls. Not theoretically hard to bake
| bamboozled wrote:
| It googles models suck
| anthonypasq wrote:
| Personally I think Apple should have acquired them. if you
| could burn a gemma4 class model into an iphone and actually get
| extremely low latency and low battery usage it would feel like
| the future IMO. even if it means you wont get frontier
| intelligence, there might actually be incentive to buy a new
| mobile device every year again.
| bsaul wrote:
| That's actually a really good point... There's currently zero
| incentive to buying more hardware, and that's one very good
| reason do have a new one.
| sebular wrote:
| But this is already happening with iPhones. Apple is
| touting on-device AI and only the latest phones offer the
| full capabilities. Newer phones will be able to run better
| models, so the incentive is there as soon as someone makes
| the killer app that only makes sense when the model is
| running locally on your phone.
| amelius wrote:
| > as soon as someone makes the killer app that only makes
| sense when the model is running locally on your phone.
|
| I expect this to be around the time when we're finally
| ready to travel to Mars.
| superb_dev wrote:
| From what I remember, these chips are not mobile size yet
| bradfa wrote:
| A small model would be. I think that's more the point. It's
| definitely not SOTA but it's fast and energy efficient and
| local.
| mdp2021 wrote:
| > _A small model would be [mobile size]_
|
| A ~30mm side for the HC1 tech for an 8b model (still
| unclear the planned HC2)?
| teaearlgraycold wrote:
| Is that analogue or are they baking floating points into
| the silicon?
| AlotOfReading wrote:
| It's entirely possible they're using something like block
| floating point, where most of the hardware is simply
| fixed point. AMD's NPU does this, for example.
| wmf wrote:
| Nope, a small model would be larger than the whole iPhone
| SoC.
| adgjlsfhk1 wrote:
| I don't think this works out from a cost/silicon perspective.
| Small models already run pretty well in software (since the
| weights fit in cache) and big models require silicon area
| proportional to the size of weights. On a mobile device
| putting a chip like this is competing directly in BOM and
| power against a whole lot more l3 cache, and the l3 cache
| makes everything faster
| teaearlgraycold wrote:
| My question is what changes about LLM use cases when you're
| getting 1000 tok/s? Models in silicon might dramatically
| change how we think about them.
| RussianCow wrote:
| That likely isn't as relevant for on-device iPhone usage
| as it is for Real Work(tm). I won't notice the difference
| between 50tps and 1000tps when asking Siri a question.
| spijdar wrote:
| I don't know. As others have said, the Taalas chip wasn't
| small, or particularly low power, so it's hard to
| "imagine" what that tech in an cell phone chip might look
| like.
|
| But if the basic premise of "good enough LLM at insane
| throughput" holds, I think it could qualitatively change
| local uses of LLMs. At a certain speed point, you're able
| to move from request -> response to a cascade of tool
| calling and "subagents", which could allow a small model
| to be much more useful, if provided with a lot of local
| data and tool calls.
|
| That said, this is assuming you could stuff a "good
| enough" model into a phone with Taalas-like technology.
| The Taalas tech demo was an 8B parameter model and
| required hundreds of watts (IIRC) to run. The efficiency
| was good given the speed (as I understand), but it's not
| clear at all that the approach scales small enough to be
| a sensible coprocessor on an iPhone or whatever.
| intrasight wrote:
| Box that plugs into my desktop would be fine. Or perhaps
| in SSF form factor.
| manmal wrote:
| > At a certain speed point, you're able to move from
| request -> response to a cascade of tool calling and
| "subagents"
|
| That also needs server class hardware though. A phone
| won't happily service the insane amount of IO, compute,
| and network that this cascade would require.
| retatop wrote:
| But wouldn't higher tps allow for more reasoning or other
| hidden processes, potententially making a smarter model?
| dabbz wrote:
| This is my thought as well. Models have to be intentional
| about which tokens they burn because there's a real lag
| time. If you can just fork out 10 different reasoning
| sessions at once with no regard for token waste/lag, you
| can compensate a smaller model with just doing more at
| once with it. No idea if this is reasonably true though.
| manmal wrote:
| I think that only works if you have checkpoints where all
| that reasoning can be checked against reality. Otherwise
| you get an army of armchair experts. LLMs are hilariously
| bad at home improvement advice btw, where reasoning alone
| won't get you far.
| nvme0n1p1 wrote:
| That order of magnitude could be the difference between
| "the users wants me to open the notes app, let's open it"
| and "I've scanned all your notes before you could blink
| and found what you're looking for".
| p1esk wrote:
| If Siri is using a 3T model in high reasoning mode to
| answer your question you will.
| in_a_society wrote:
| The best way I can explain it is that it's the same
| feeling when I upgraded from 56k dialup to cable
| broadband.
| HDBaseT wrote:
| In the case on on-device/self-hosted LLMs. You ask your
| agent to implement xyz feature 10 times and use a model
| to compare the outputs and combine the best results.
|
| Raw intelligence becomes slightly less important when you
| can iterate and improve automatically. You can still
| claim it was "one shot" even when 30 different
| implementations were made then combined.
| manmal wrote:
| Problem is, there exists no judge model that will really
| pick the same winner that you would.
| QuiDortDine wrote:
| Did you use chatjimmy? It's somewhat terrifying to use
| when you think of the potential results with a better
| model.
|
| Ok, real life example: I now spend most of my time, as a
| developer, waiting for the agent to do its thing (after
| careful prompting, I'm also thinking about work stuff,
| don't worry I'm not useless). What if it gave back the
| same excellent results, but instantaneously? Why, then, I
| certainly would become the bottleneck. So, quite
| possibly, my last work task would be to plug this agent
| directly into the ticket system where the domain experts
| input their feature requests. Maybe we still need 1
| developer out of 100, to coordinate releases and all that
| (ok, say 1 out of 10).
|
| But that's not taking things far enough: why do we need
| these domain experts at all? Our pitch is clear, and all
| software-enabled, though it took years to develop. We can
| just have the clients express their concerns to the AI,
| directly or indirectly. Have multiple lighting-fast
| agents with different roles (refactoring agent, new
| features agent, debugger agent, domain expert agent,
| etc.). So we fire everyone, maybe keep 1 product owner /
| devops to keep the trolls out. The cost is still probably
| 100 times less than it used to be (beyond the initial
| cost of acquisition of the magic machine or whatever).
|
| But one of these clients, surely, will realize that these
| 10 years of manual and slowly-automated development can
| now be emulated in very, very little time. Why not just,
| say, take screenshots of the entire app and feed them
| into the magic machine? Why, this way, they could have
| the service for a tenth of the yearly cost, forever!
|
| And then the economy implodes.
|
| I'm not saying it's THE most likely version of things,
| I'm saying that at a certain level, quantity (or rather,
| speed) is a quality all its own. And this new quality
| might change the world. Let's hope it's for the better!
| wsintra2022 wrote:
| I think this reads like Ray Kurzwheil (sorry not able to
| spell that off top of my head, that bloke who wrote that
| book about the future) .. But yeah very dystopian and
| totally realistic. Not if but when..
| QuiDortDine wrote:
| I LOVE Kurzwheil! Thank you for the compliment, I'm very
| far from having his writing skills. But yes sci-fi is
| looking more and more like, well, sci.
| xur17 wrote:
| I'm not sure inference speed is always the slowest thing
| for me right now. The agent is running tests, loading
| webpages, etc, which all take time. I don't know if a
| fast agent would speed things up in all cases.
|
| That said, it obviously depends on the project.
| jodrellblank wrote:
| > " _The agent is running tests, loading webpages, etc,
| which all take time_ "
|
| A frustrating vision of the future would be when we've
| been asking for faster loading lighter web pages for
| years and then companies start caring about it and
| improving it not for us humans but for LLMs.
| evilduck wrote:
| It's already kind of that way with MCP servers popping up
| everywhere. The JIRA MCP server is like a couple orders
| of magnitude faster to work with than the website itself.
| manmal wrote:
| That's their API with extra steps, or am I missing
| something? That was always faster.
| andersonpico wrote:
| They finally cared about clear requirements and
| documentation when that meant getting rid of devs.
| layla5alive wrote:
| That happened at corpo work for each of: * Build times *
| CI latency * Developer tooling * Documentation *
| Modularity
| bdangubic wrote:
| > I now spend most of my time, as a developer, waiting
| for the agent to do its thing (after careful prompting,
| I'm also thinking about work stuff, don't worry I'm not
| useless).
|
| you need to launch 10-15 more terminals, who is waiting
| these days? :)
| QuiDortDine wrote:
| You sound like my boss! I'm not really into the whole
| "burnout" thing though.
| bdangubic wrote:
| how can you get burned out just watching the work being
| done for you?? :)
| pastel8739 wrote:
| This is the same pitch that people make about AI today.
| Speed isn't the differentiator, quality is
| Certhas wrote:
| So for every work produced by AI have ten separate agents
| review it thoroughly.
| sroerick wrote:
| They are both the differentiator.
|
| AI previously provided speed but not quality. As soon as
| quality reached an acceptable threshold, the speed became
| the reigning factor.
|
| In my opinion the quality is still much lower, but speed
| means the cost is significantly lower also.
| TeMPOraL wrote:
| AI is already fast enough that human is a bottleneck.
| Hell, _typing speed_ became a bottleneck like it was
| never before.
|
| I mean, if an agent can do half-decent work in less time
| than it takes the user to prompt them (and "user" in this
| context is a fast touch-typist like most programmers
| are), it's obvious it's not the agent that's the
| bottleneck anymore.
| tripzilch wrote:
| Because finishing someone else's (or _something_ else 's)
| "half decent work" to the point of "actually decent"
| becomes the bottleneck.
|
| This has always been the case for human project
| management, and LLMs just aren't at that level yet.
|
| It's more like everyone is speed running to how fast they
| can convince others that "half decent" is good enough.
| And for sure, newer models of LLM seem to be getting
| better at _that_.
| TeMPOraL wrote:
| > _It 's more like everyone is speed running to how fast
| they can convince others that "half decent" is good
| enough._
|
| But that's what Agile is all about, isn't it? We've been
| speedrunning delivering increasingly smelly shit at
| increased velocity ever since SaaS became a thing,
| because ubiquitous Internet access is what allowed our
| industry to adopt the "lob feces over the fence for users
| to deal with" release model.
|
| AI does speed that up, true (though since the market -
| and management - didn't catch up with it yet, we have a
| brief moment where we can use AI to increase quality
| while keeping usual delivery rate.)
| tiborsaas wrote:
| Speed will be one of the killer features once you get
| closer to instant speeds of 300ms. Just remember what
| changes were made possible simply by upgrading from ADSL
| to broadband.
|
| If inference speed goes up, I can launch the same query 5
| times, evaluate the best result and proceed from there.
| Of course, evaluation is also instant, so in seconds I
| can get a near perfect solution. Or maybe 10 and I can
| pick what I like the best.
| manmal wrote:
| Errors compound, and making 1000 wrong decisions per
| hour, will not result in something useful. Maybe you've
| tried setting up guardrails for good design or
| architecture at some point? I think it's simply not
| possible to do that.
|
| It would certainly be an accelerator for people who know
| exactly what they want. And it would remove multi
| tasking, which I'd appreciate.
| Ericson2314 wrote:
| If your task has incremental rewards/feedback, you can
| push the "intelligence rate" simply by sampling the
| reward function faster. That's not fake, even if it not a
| substitute either.
|
| This is the "dumber but honest person that works harder"
| phenomenon, vs "lazy genius".
| gf000 wrote:
| That's a good way to put it, but still my experience is
| that worse code bases are non-linearly harder to maintain
| and improve in the future, software tends to break down
| without a good enough base.
|
| Sure, in the future full rewrites and stuff like that
| will be just another "throw money at it" problem, but
| fundamentally software can get arbitrary complex and we
| barely know how to write large, maintainable code bases.
|
| Nonetheless, I think testing (and maybe proofs) will have
| its long-awaited time to shine, as being _the_ "reward
| function".
| Ericson2314 wrote:
| I totally agree with you on the first bit, but I also
| think that I am way better at _deciding on how to
| refactor_ code bases than the LLM is.
|
| Right now, I put models in low thinking mode during my
| refactors and hate waiting. I would much rather have a
| faster model that that maybe was slightly stupider, and I
| would wait far less long between prompts where it needs
| my valuable input.
|
| Models that are dumb, but humble and fast, can be fine.
| momojo wrote:
| I don't have a great answer but you pose a great
| question.
|
| Obviously a CTO is not going to walk away from the
| technology just because it's not good enough. That much
| more incentive for someone to create a powerful enough
| harness that can direct that power safely and
| productively. Like a nuclear core, we'll need to come up
| with the graphite rods and water tank. And if tokens are
| essentially free, why _not_ , for every million tokens,
| spend 10x tokens on code review, testing, etc?
| manmal wrote:
| I do spend 5x more tokens on planning and reviewing, than
| for implementation. But architecture is still nothing I
| can delegate.
| visarga wrote:
| AI helps you but also your competition, and gets factored
| in by investors while customers can use it to find better
| deals. The whole market is different even if a company
| did nothing.
|
| Whatever you can cheaply do with AI is not a moat, if
| there is profit in there there will be quick imitation
| and competition will eat away those profits.
|
| Models can be replaced easily, harnesses & AI tools too.
| And if cloud inference gets too expensive there are local
| models keeping the cloud prices hard capped.
|
| Probably AI won't make anyone very rich.
| IOT_Apprentice wrote:
| I tried it. I asked where Bruce Lee was born. It stated
| he was born in Hong Kong. I challenged it and it went
| further naming a hospital there. I stated he was born in
| San Francisco and it apologized and then said his father
| was a missionary traveling in America, which was also
| wrong. Bruce's father was a famous Cantonese Opera singer
| and actor.
|
| This model had zero information right, while being fast
| in responding.
|
| Unacceptable.
| selcuka wrote:
| It gave the correct answers to both questions for me:
|
| > Bruce Lee was born in San Francisco, California, USA on
| November 27, 1940.
|
| > Bruce Lee's father was a Chinese opera singer
|
| That being said, this is not a good test. It is a
| language model (a very small one), not an encyclopedia.
|
| ChatJimmy interface is just a tech demo. Without tool
| calling functionality we can't expect it to be factually
| correct.
| logicallee wrote:
| if it's baked into silicon how can you two get different
| answers?
| v9v wrote:
| It still works the same way other LLMs do, by outputting
| the probability distribution over the possible
| completions (The weather is ... (sunny (50%), cloudy
| (50%))). Then the next token is sampled from this
| probability distribution (in our example the next word
| could be "sunny" or "cloudy" equally likely), which can
| result in different outputs every run.
| logicallee wrote:
| Could the model or algorithm be changed to make it
| deterministic somehow? It could help a lot if there were
| reproduceable outputs from deterministic baked-in
| silicon.
| Tuna-Fish wrote:
| You can make any LLM deterministic by dropping the
| temperature hyperparameter to zero.
|
| This will generally make them suck, though, a little bit
| of randomness is necessary for proper function.
| fwip wrote:
| You can also use a fixed seed for your prng. A hash of
| the input text (up to the current turn) should do.
| fph wrote:
| But since it's so fast you can just ask it 100 times
| where Bruce Lee was born, and statistically you'll get
| the correct answer. We could call it "mixture of idiots".
| /s
| TeMPOraL wrote:
| That's not what speed is useful for.
|
| I just pasted your comment and its whole inheritance
| chain to it, started my comment, and asked to generate a
| total of 9 completions, 3 from each of {current & next
| word, current paragraph, current paragraph + rewrite the
| entire paragraph}.
|
| Half of the answers were perfectly good (ironically, not
| the "next word" ones!), but the important bit, they came
| back near-instantly ("Generated in 0.024s - 14,163
| tok/s", the page says). Slightly more powerful model
| while keeping this under a second, and this could easily
| become a qualitatively different form of
| autocomplete/text suggestion. Running in the background
| every couple keystrokes, or every time user stops typing
| for more than 500ms.
| logicallee wrote:
| >That's not what speed is useful for.
|
| >I just pasted your comment and its whole inheritance
| chain to it,
|
| Good idea. Only problem is _it doesn 't work_. I just did
| the same thing with exactly this prompt:
|
| >did the user IOT_Apprentice participate in the thread
| below and if, number and quote all of their comments.
| Only just number and quote the comments or write "Did not
| participate", do not add any commentary. Quote any
| comments by this user verbatim, exactly as input. Thread:
|
| followed by pasting the thread[1]
|
| And received the answer "IOT_Apprentice did not
| participate in the thread."[2] in 0.001s, even though
| they have literally the last comment in my quote and it's
| clearly legible.
|
| It's particularly insidious because the understanding and
| thinking that is required to follow my requested answer
| format exactly is substantial - so based on the fact that
| it gets the _format_ right and clearly understood the
| assignment, I would be inclined to believe that it would
| also be correct!
|
| So to use your example, it's not just autocomplete, it's
| autocomplete that confidently returns "No matching
| results" in 0.001 seconds, even though there is a search
| term matching what you put in, right in the prompt itself
| that was sent to it. That is much worse than useless.
|
| [1] prompt: https://ibb.co/CKVmRvtd
|
| [2] result: https://ibb.co/BKdRKmyD
| tliltocatl wrote:
| Using LLMs for information retrieval is the most stupid
| thing one can do. Especially when old methods work much
| better.
| LarsDu88 wrote:
| Massive economic simulations with thousands if not
| millions of agents to front run the global economy and
| stock market.
|
| Fully interactive realtime NPCs in videogames at scale.
|
| Recommender systems that simulate individual consumers.
|
| Crazy shit
| manmal wrote:
| About your first example, isn't the butterfly effect
| preventing this from being useful? One agent in your
| simulation decides to sell, and starts an avalanche, that
| won't happen in reality?
| bostik wrote:
| When you run tens of thousands of simulations for complex
| economic models, you actually _do_ want to see the
| extreme outliers too. I can 't recall who said it, but in
| finance the interconnected incentives make so-called
| Black Swan events much more likely and frequent than
| models or theories can comfortably account for.
|
| In a way... when it's finance, they should be maybe
| called Gray'ish Swans?
| LarsDu88 wrote:
| Run the sim many times... faster than it can run on
| actual humans and compute a probability density for
| specific events.
|
| Better yet use it to dimulate counterfactual phenomena
| like market manipulations ypu intend to enact...
| TeMPOraL wrote:
| It's pretty clear if you see what's happening on current
| phones.
|
| Autocorrect that works. Reply suggestions that almost
| work, just need to be tad more accurate (probably more of
| a data access issue than model) and a tad faster to look
| completely seamless. Screenshots with automated text
| detection and OCR and automatic interpretation (different
| suggested actions for when something on the picture looks
| like a web link, phone number, postal address, e-mail, or
| QR code, or an event poster). That's just a fraction of
| things I saw showing up on my Samsung phone over the last
| 6 months.
|
| For over a year now, you could get a _much_ better
| autocorrect and spell /grammar check, _and_ a translator
| all in one, if you just pasted your text to a frontier
| model and asked it to check for errors or translate into
| target language. Now imagine being able to go through a
| round of such checks in a 1 /100 of a second. You could
| have this running every keystroke, and suddenly the
| inline autocorrect/checks would not suck anymore.
|
| Auto-linkifying that can correct for typos and doesn't
| need careful regex tuning because it understands from
| context what is _meant to_ be a link or not. That 's just
| one of many obvious things possible once you get local
| models running fast enough. Tip of an iceberg, and the
| first step to imagining all the other potential uses is
| to let go of the two mistaken beliefs people hold on to:
|
| 1. That LLMs are about written language. They're not;
| ever since "multimodal models" became a thing,
| tokenization extended to visual and audio space, and now
| textual and visual and aural inputs are all just regular,
| first-class tokens.
|
| 2. That chatting with the models is the only optimal way
| for end users to interact with AI. That's just
| artificially limiting yourself to the space of chat-based
| UI.
| bastawhiz wrote:
| The weights might fit in cache, if you're using a small
| model. If you wanted to have a 20B+ parameter model, that's
| just going in RAM. You could put more RAM in the device and
| pay the perf cost or have a dedicated chip. Most devices
| already have a dedicated chip, this just changes which
| silicon you're spending the money on.
| wmf wrote:
| That math doesn't really work.
|
| 8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2
|
| 8B model (Taalas) = 4 GB ROM = ~800 mm2
| dboreham wrote:
| Works great from a press release perspective though.
| trebligdivad wrote:
| What, even if it means you can run models without relying
| on the currently backlogged DRAM production?
| adgjlsfhk1 wrote:
| The size of model we're talking about running doesn't
| need much if any dram.
| trollbridge wrote:
| The chatjimmy demo is using a model that needs 6-18GB of
| VRAM. That's not exactly trivial.
|
| I could see it being feasible to get a Qwen-3.6-27b type
| of model done on something like this. Qwen-3.6-27b at
| 18tok/s would be a game changer.
| adgjlsfhk1 wrote:
| right, but that's a reticle size chip. to put something
| in a phone it has to be ~10-30x smaller
| Melatonic wrote:
| The Taalas chips are not physically small. And part of their
| secret (if you look at the design) is just locating a bunch
| of memory soldered on the edges ( I belive higher amounts of
| SRAM ? )
| chorizo wrote:
| Baking the base models on to ROM makes a lot of economic
| sense. SRAM for the KV cache & fine-tunes, not so much.
| Sure you'd get incredible speeds but it's not scalable from
| a die-size or cost perspective.
|
| Rather base model on ROM + KV cache on DRAM is much more
| scalable. Also this would work great for edge devices that
| have a 2-5 year lifecycle.
| adrianN wrote:
| It is my understanding that just baking the model itself
| into silicon only gives moderate gains because memory
| bandwidth remains a bottleneck.
| chorizo wrote:
| The big benefit is ROM cells require fewer components
| than DRAM. So the chips would be tiny, dense, cheap and
| consume far less power.
| klodolph wrote:
| I thought DRAM was pretty dense already. Is mask ROM that
| much denser?
| chorizo wrote:
| Yes, each rom bit can be a transistor or even a diode
| with a decoder circuit. Simplest Dram cell is
| capacitor+transistor - and you need a clock, refresh
| circuit etc.
|
| Someday, I imagine model weights could even be encoded as
| analog resistors (memristors or similar) for even greater
| density
| ReactiveJelly wrote:
| Hm. I wonder how many relays I'd need to make a physical
| MNIST classifier. That'd be dope
| Tuna-Fish wrote:
| It's not even just that. If you just built the rom chips
| separately and swapped them for the RAM of a normal
| accelerator, it would not help at all.
|
| The trick is that every compute element in their system
| has it's own small pool of ROM, instead of putting all
| the ram behind a common pipe. ROM is just used because
| it's the densest kind of memory that can be fabricated on
| the same process as their logic.
| lelanthran wrote:
| Did you try using the the talaas chat? Something stupid
| like 18k tokens/second.
|
| Think it's called Askjimmy or similar.
| jgilias wrote:
| Oh boy, thanks for sharing this, truly mind blowing. It
| was chatjimmy.ai
| TeMPOraL wrote:
| What is the model they're using there though?
| Interrogated, it claims it's a BERT variant and has
| capabilities around GPT-3 and below GPT-4.
|
| (Not that I believe it, it writes too well for GPT-3.)
|
| Hosted frontier models from two years ago would be much
| faster today, too.
| egorfine wrote:
| They run Llama 3.1 8B.
| onion2k wrote:
| _Baking the base models on to ROM makes a lot of economic
| sense._
|
| Less so for consumers though, because it'd mean the phone
| is out of date in 3 months when a better model comes
| along.
| ronsor wrote:
| It's a perfect reason to get consumers to buy a new phone
| every year again! They got bored of the camera.
| miohtama wrote:
| This stopped working some years ago.
|
| Now you sell the same phone with higher price tag.
| ben_w wrote:
| Right now the models are doubling in performance (by the
| METR time horizon metric at least) every 4 months, so 3
| doublings in a year; conversely, I hear (not my field) it
| takes around a year to make a prototype IC and another
| year to turn that into mass production, i.e. if the next
| (late-2026 model) iPhone has a chip like this, it will
| likely be with, at best, a late-2024 set of weights. I
| think you can get open-weights models today that have
| performance equivalent to the SOTA-late-2024 while
| fitting in the RAM of a (high end) 2025-26 phone.
|
| At some point the music will stop on training bigger
| models, and when that happens it will make sense to have
| ROM weights (or 100% analog circuits given how noise-
| resistant LLMs are), but we'll know when that is because
| the investment bubble funding the training of new models
| will have burst.
| close04 wrote:
| At some point it's got to be good enough for the normal
| "phone stuff" that appeal to most users. So they wouldn't
| suffer from FOMO because they didn't wait for the next
| model. Every phone gimmick went through the same
| evolution curve until it passed the "good enough" point
| and eventually plateaued.
| ben_w wrote:
| Yes, but irrelevant. While these models are improving at
| the present rate, the manufacturer can save money at no
| loss of feature-bullet-point-on-website by letting you
| download a model after you bought the thing and running
| it on normal hardware.
|
| The rate of change to the models has to be slower than
| the hardware roll-out to be worth a hardware solution. If
| "good enough" happens before then, that just means the
| user gets a software solution.
| close04 wrote:
| You might be right but hard to tell without analyzing
| costs and benefits. Is a cutting edge model for phone
| stuff worth the slower performance and battery drain for
| example?
|
| The rate of change by itself doesn't tell you the whole
| story because of costs and diminishing returns. So what
| if your model is twice as good if it's 10x the cost and
| it saves you 1ms? Everything else about phones reached
| "good enough for a phone" levels in years, and then got
| minimal generational improvements.
| dgrabla wrote:
| wait, analog circuits? can you elaborate this?
| inigyou wrote:
| Circuits that aren't restricted to two particular voltage
| levels.
| ben_w wrote:
| Transistors can be used to amplify signals, they are not
| limited to acting as binary switches. If you use analog
| rather than digital, using transistors in this way means
| you can replace however many transistors it would have
| taken for multiplying two n-bit numbers with just one; I
| understand capacitors can be used for accumulation, but
| don't know how many additional components that needs as
| I'm an electronics noob.
|
| The reason we don't do this in general (any more) is that
| for long chains between input and output it has been much
| too difficult to avoid accumulation of errors. LLMs
| happen to be extremely resilient to errors like this,
| which is also why we can use e.g. 4-bit weights.
| chorizo wrote:
| And the customers can wait for the new phone released
| next year. These are edge models - the average customer
| doesn't need the latest frontier model. Just needs to be
| good enough for the features you promised.
| otabdeveloper4 wrote:
| That's a software engineering problem. They just need to
| figure out how to fine-tune for alignment and tool usage.
|
| That's the only thing the normie consumer cares for
| really.
| fsloth wrote:
| I don't think average user _needs_ to solve frontier
| challenges. "Call to Jane", "turn on the lights" and
| "what's the weather this afternoon" is more like it I
| would guess.
|
| Ofc if the model has some critical bugs that's another
| matter.
| dzhiurgis wrote:
| Your examples worked on phones for over a decade.
|
| Maybe baking in a model that is "certified" to have some
| unconditioned truths + rest is pulled from external
| models/store could make sense. But AFAIK that doesn't
| exist and I'm not sure it can possibly be made. Perhaps
| society as a whole at least can work on an open corpus of
| training data, but I'm not holding my breath on this.
| logicallee wrote:
| >Your examples worked on phones for over a decade.
|
| Nope. And not only not a decade ago, right now.
|
| If you have an Android or iPhone, you can give it clear
| and easy to understand instructions that Gemma 4 could
| complete[1] if it had tool calls on it, and that 100.00%
| of Claude, ChatGPT, Grok, Kimi, you name it, could
| understand and all complete if they had the access.
|
| The phones will fail to complete it. I just tried Siri. I
| said "hey Siri", waited for Siri to come up, and then I
| asked one of the exact sentences you replied to: "what's
| the weather this afternoon?" It thought for around 20
| seconds, and said "Something went wrong. Please try
| again."[2]
|
| I have Wifi, I have mobile Internet, I have free storage
| space, I have up to date software. What went wrong is
| that phones have never properly connected agents, not ten
| years ago, not last year, not this year, and probably not
| next year.
|
| But don't settle for what Google could do in 1999 by
| hotlinking the keyword "weather" in any query to the
| weather being shown in the results.
|
| Tell your phone (any phone): "Please call back the last
| number that called me that is not an unlisted number,
| regardless of who it came from."
|
| 0 out of any phone will complete that today, tomorrow, a
| year from now, five years from now, ever, because phone
| makers are not going to let them do that.
|
| Meanwhile, 100% of all frontier agents could complete it
| _if_ they had tool calls on the phone. Which they don 't,
| and won't ever, thanks to the duopoly.
|
| Okay, that's a bit dismissive, I would love to be wrong!
|
| [1] after any voice recognition to text - which does work
| really well on both Android and iPhone! [2] screenshot:
| https://ibb.co/21rtDnfV
| narism wrote:
| I'm on the IOS 27 beta and Siri did those two tasks
| (weather/phone) flawlessly. It's a lot better than it
| used to be.
| logicallee wrote:
| Thanks for trying that! Very interesting.
|
| Can you say this to it: "Hey Siri [wait for it to come
| up] - please send me an email with the temperature right
| now so I have it for my records." and see if it can
| complete the task without any backtalk or
| misunderstanding, and if you get exactly what you asked
| for. (It's a really clear request.) Should be 1
| statement, no clarification, conversation, random search
| results, ("Here's what I found!"), etc.
|
| A normal frontier model can do that - or Siri can do it
| if it is properly connected to Claude, ChatGPT, Gemini,
| Grok, or any other frontier AI - but previously it was
| never properly connected.
|
| If it can do this task, I might have to look into this
| again. It counts as a success if it sends yourself any
| email with the current temperature and you actually get
| it (it can include whatever other text in the email), and
| a failure if it talks back, says "here's what I found",
| says it can't, asks you any question, sends you an email
| that doesn't actually contain the current temperature,
| just reads you the temperature and then asks if you want
| it to send an email, etc. Should be 1 shot.
|
| let me know if it works!
| narism wrote:
| It brings up a preview of the email and you have to tap
| or tell it to send it from there but otherwise it worked
| for this as well.
|
| Subject: Current Temperature Body: The current
| temperature is 27degC in <my city>.
| logicallee wrote:
| thanks! useful.
| fwip wrote:
| In my experience, they had a lot of stuff working well in
| the first few years they rolled out the home voice
| assistants - Alexa, google home, etc. But for whatever
| reason, they've spent the last eight(?) years silently
| breaking things that used to work. Stuff like audiobook
| playing, music alarms, or even messaging people.
|
| Once they started seeing useful (if niche) functionality
| as a cost center, there wasn't really a world in which
| these could usefully exist. Their big bet now seems to be
| that LLMs will lead them to profitability - but whether
| that's from increased data harvesting, cheaper
| integrations, or because it'll be useful enough to charge
| subscription fees, I couldn't tell you.
| gf000 wrote:
| It barely works even today, like Siri is laughably bad.
| dzhiurgis wrote:
| I mean the examples he gave definitely work. Mostly well
| I'd say as they are pretty primitive.
|
| What Siri is missing is more logical solutions and
| answers for recipes, etc (still suck even with chatgpt
| integration).
| gf000 wrote:
| Works, sometimes. But they can fail spectacularly and
| unexpectedly even on _very_ basic questions
| /instructions, like so simple that a hand-coded word-
| matching style logic could get them right 20 years ago.
| darkwater wrote:
| No, they don't work. Just asked Siri the other day
| "what's the weather tomorrow in $LOCATION" (where
| $LOCATION is a broader zone and not strictly a city) and
| the answer was the weather in a street called "$LOCATION
| Avenue" in a city 150km away.
| yunwal wrote:
| I just asked Siri
|
| "Hey Siri, what's the weather in <nearby town with a
| generic name> tomorrow" and it gave me a town with the
| same name ~800 miles from me.
| tesnorindian wrote:
| We also have ReRAM (Analog Computing), which also holds a
| promising future given its efficiency and low power.
| Though ReRAM of larger size is still a research area.
| petra wrote:
| AMD can let the SRAM be on a different chip. Maybe even
| something similar to their 3D cache. that could increase
| density to 20B[1]. They could also move from 6nm to 2nm.
| that would probably increase density by another 3x to
| 60B.
|
| Add a bunch of chips together, and you get to a server
| that can run a 800B model, very fast and probably
| significantly cheaper than others.
|
| [1]https://www.eetimes.com/taalas-specializes-to-
| extremes-for-e...
| coderbants wrote:
| I think the lifecycle for these chips could stretch far
| longer. If you're offering these models on a two year
| lifecycle, then you'd be able to stand up your top tier
| (wouldn't need to be frontier) at high speed. Run (for
| example) Kimi K3 on it and give it a brand name:
|
| AcmeAI Carbon
|
| Market it as your premier (only) model at high
| throughput. Two years later you stand up MSICs for the
| new state of the art with entirely new hardware, your
| lineup becomes:
|
| AcmeAI Nitrogen (top tier) AcmeAI Carbon (mid tier)
|
| If you just kept pushing the same model down your pricing
| tier over time you could still extract a lot of value
| from an old model, even years after it's been set in
| stone. Working on brand new code/frameworks? Pay to use
| the newest model. Working on legacy code? Use the lower
| tier models that will already know your legacy
| frameworks, pay far less and still get massive
| throughput. I've worked on a lot of government projects
| that this would be absolutely brilliant for.
|
| The other side of this is that agent harnesses are NOT
| set in stone, so even a legacy model with a knowledge
| cut-off that's years out of date can likely still be
| helped quite a bit by harness and fetch behaviours that
| are still developing rapidly. Especially at this kind of
| throughput.
| hypfer wrote:
| I'm somewhat doubtful that we will be seeing something as
| large as Kimi K3 in silicon any time soon.
|
| This tech can definitely scale up from the current 8B
| prototype, but - at least as far as my limited
| understanding of the tech involved goes - you cannot just
| ASIC a trillion weights model due to physical size
| constraints.
|
| ___
|
| Specification HC1
|
| Model Llama 3.1 8B (hardwired)
|
| Process TSMC 6nm
|
| Die size 815mm2
|
| ___
|
| So the current prototype already pushes the limits of
| what we can fit on a single die, and that is already
| likely going to limit your yield.
| A_D_E_P_T wrote:
| Kimi K3 is huge, though. Deepseek V4 Flash is a much more
| moderate model (284B total), and it works _extremely_
| well. Models of that size, and smaller, are just going to
| keep getting better and better. Presumably there 's a
| threshold below which models are not generally useful or
| competitive, but if models-on-silicon can scale up to
| just 256B, that would be really remarkable.
| momojo wrote:
| My 2 cents to for your point:
|
| - Deepseek V4 Flash is _impressively_ capable. Sonnet
| still beats it out by a thin margin, but the real kicker
| is that a typical session with Sonnet at current API
| costs is ~$2. The same session with Deepseek is 2 cents
| (ha). Its even allowed me to consider offering free-with-
| limits API usage on my own app. - Taalas (or competitors)
| have a lot going for them. If anything I feel like they
| need to join hands with these smaller model makers and
| converge in 2028
| LarsDu88 wrote:
| This is an architectural limitation that may be overcome
| by how you bake the MoE (mixture-of-experts) onto
| silicon.
|
| If you could manage a per-die expert somehow and keep the
| expert routing gate relatively fast (through an
| interposer interconnect or doing wafer-scale Cerebras
| type shit) you don't need to keep the whole thing on the
| same die. Small dies with one expert per die on an
| interposer, and a very tiny router might be sufficient.
| to11mtm wrote:
| There's in-betweens; '1T SRAM' or eDRAM.
|
| Of course, 1T SRAM isn't really SRAM, but my
| understanding is it doesn't require external refresh like
| eDRAM, is a bit easier to fab on-die than eDRAM, and is
| half the mm2 per Megabit compared to real SRAM (15% more
| die size than eDRAM)...
| selcuka wrote:
| Their PoC chips are big, but then it's ridiculously fast
| (have you seen chatjimmy.ai?). Also they must be holding a
| bunch of patents.
| aenis wrote:
| Its a cool demo, but its gpt-3.5 level stupid, or worse.
|
| edit: Ok, I will self-apologize. Its apparently a 3B
| model. Mighty impressive for what it does.
| selcuka wrote:
| It's a quantised 8B model (Llama 3.1 8B to be exact).
|
| [1] https://taalas.com/the-path-to-ubiquitous-ai/
| teiferer wrote:
| How time flies. Just 3.5 years ago, gpt-3.5 was touted as
| almost AGI, we're all going to be replaced by machines
| and worst case they will kill us all. And here we are,
| not much later, and it serves as the benchmark for
| "stupid"..
| deaton wrote:
| Sure but the first step to having something that is
| physically small, small enough to cram into an iPhone, is
| to have something that, at first, isn't.
| makeitdouble wrote:
| Slightly besides your point, but it's interesting how many
| here naturally ponder about how the current winner could or
| "should" keep winning, instead of how another company could
| become a competitor by doing the more clever thing the
| incumbent isn't thinking about.
| krisoft wrote:
| It is not a "should". At least not in the "we wish it were
| so" sense.
|
| It is more that there are multiple reasons why this idea
| (burning an LLM into silicone and deploying it into a
| device in people's pockets) requires huge piles of cash and
| the kind of engineering chops only a few company posesses.
|
| Of course i would like it if a small upstart would do this,
| but it doesn't seem likely as a posibility. They won't have
| the funds to fab the IC. They won't have the funds to train
| and validate the model before burning it into silicone.
| They can't absorb the risk of the first tape out going
| wrong. They can't absorb the risk of the model being faulty
| in some subtle way. They don't have a device to integrate
| the IC into. They won't have the funds to develop one. If
| they somehow would make a device they don't have the
| marketing and sales channels built out to get the device
| into people's hands in sufficient numbers to justify the
| development cost.
|
| Basically this idea feels ruinously expensive. Apple has
| deep pockets, they already have working well-regarded
| phones, and an ethos of privacy preserving innovation. This
| is why this idea feels well suited for them and not many
| others.
|
| Do i want the winners to keep winning? No. But not many
| others can pay for a moonshot crossed with a manhattan
| project. They just can't.
| ricksunny wrote:
| Yes, it's an interesting register (sorry for the
| claudism; blame lesswrong-weighted training) for the use
| of the word 'should'. I agree with your assessment and it
| is rarely articulated. Sometimes I think that the HN set
| is abused by big tech both from above on the employer
| side and the consumer usage side (all the T&C's, VC
| incentives and M&A taking away once-good-things). So they
| adopt the only sliver of agency-salving language
| available, like 'big company that I have no scope over
| should X'.
| makeitdouble wrote:
| Good point. IMHO we're thinking of too much in single
| entities doing everything.
|
| Right now it's kinda the world we live in, with Apple or
| Google doing the total vertical integration from chip
| design to retail shops, but it doesn't have to be that
| way.
|
| It could be done the traditional way with for instance a
| joint venture receiving funds and expertise from several
| players in each of the field and collaborating with
| external companies to get to the final package.
| freekh wrote:
| It would be cool if the future was a standard fairphone like
| module system where you could replace the model chip when you
| felt like it without having to shell out 1-2k $$$s for a new
| phone
| dzhiurgis wrote:
| Its wild but if chip is something like $30 and provides
| frontier intelligence then just throwing them away every 3
| months isn't that big of a deal when a lot of us pay $50 to
| $150 to $1.5k per month on AI tools.
|
| I don't think it needs to be on phone per-se. It can keep
| chugging in cloud - plenty of people use cheaper older
| models.
|
| And I suspect the growth will slow eventually making taalas
| interations slower.
| koiueo wrote:
| > if you could burn a gemma4 class model into an iphone
|
| ... you would still have a mediocre phone with half-assed
| barely working features driven by locked down proprietary
| software
| wolttam wrote:
| It's a terrible moat. You etch the silicon then nobody wants to
| run it in 6 months because models have advanced that much
| further.
| speed_spread wrote:
| If a model is good enough today, it's still gonna be good
| enough in a year. Except you'll be able to serve it 1/100 of
| the price. Or 100x the speed.
| wolttam wrote:
| I think we will eventually reach a point where this is the
| case, but at the moment it seems like you can throw
| virtually any non-trivial use-case at a model today and end
| up being more satisfied with the results that a model
| tomorrow gives.
|
| I may just be closed minded as to what use-cases we have
| that current models are _truly_ "good enough" (i.e. won't
| be dissatisfied when comparing results of today's model to
| tomorrow's model)
| nine_k wrote:
| Not so if it's embedded in something smart enough for its
| intended purpose.
|
| Think vision, spatial reasoning, speech synthesis, even some
| speech analysis. Think self-driving cars (and drones) that
| need 10x less power for the brain, and can think at 10x
| situation per second.
| anigbrowl wrote:
| This is only true for people who are solely focused on
| performance. There is absolutely a market for acceptable
| performance combined with predictability.
| teraflop wrote:
| True, but predictability cuts both ways.
|
| We're all used to having to constantly update our browsers
| and phones to keep up with the security arms race. If a
| frozen model can't be updated, it will predictably remain
| vulnerable to any "exploits" or idiosyncratic quirks that
| people discover over time.
|
| Let's say, as somebody suggested in another comment, that
| you buy 100,000 of these chips and deploy them to run fast-
| food drive-thrus. And then somebody discovers the model has
| a fondness for goblins[1], and if you role-play
| convincingly enough, you can get it to accept payment in
| shiny buttons and rodent skulls instead of cash.
|
| What do you do then? I guess your options are to try and
| fix the behavior with a better prompt, or put some kind of
| filter in front of the model to catch attempted exploits.
| If the filter is cheap and dumb it probably won't work well
| enough, and if you use another model as a filter, you've
| negated the cost and speed benefits of putting the first
| model in hardware.
|
| Of course the real answer is to just never expose the model
| to situations where an adversarial input could possibly
| lead to an undesired output. But that drastically limits
| what you can do with it.
|
| [1]: https://openai.com/index/where-the-goblins-came-from/
| noisy_boy wrote:
| > Of course the real answer is to just never expose the
| model to situations where an adversarial input could
| possibly lead to an undesired output. But that
| drastically limits what you can do with it.
|
| Does it though? Isn't that what CPUs are, very fast-not-
| so-clever computing brain surrounded by layers that
| protect it?
| anigbrowl wrote:
| I see your argument but your example seems highly
| contrived. I can't think why you'd want to use something
| like this for something as dynamic as takeout ordering,
| where you might have to deal with bad customers, supply
| chain breakages, public health recalls, or any of many
| other probabilistic events.
|
| I think it's far more likely to see them used in safety
| critical applications where you need a capable model that
| can run on low power and doesn't have multiple layers of
| operating abstractions between the model and the
| hardware.
| fwip wrote:
| What safety critical applications would be a good fit for
| LLMs?
| anigbrowl wrote:
| I'm not thinking of language models specifically, but
| large neural networks in silico. I feel like a 27B
| parameter model would likely be capable of flying and
| landing an airliner, for example.
| twobitshifter wrote:
| OTOH, people get a new iPhone every year and they are ok with
| it.
| nomel wrote:
| How is that in any way related to a consumer device? This
| method doesn't reduce physical memory requirements, so
| still results in _huge_ die area. This isn 't a for-end-
| user thing, probably for decades.
| twobitshifter wrote:
| Ok, how long until nvidia gives us a new GPU?
| nomel wrote:
| I don't follow. How is that related? GPUs don't have
| fixed memory. You don't throw them away when you want to
| load a new model.
|
| NVIDIA will probably give us a new GPU when someone
| competent in the free market decides they want
| wheelbarrows full of money. Unfortunately, AMD is
| entirely, incomprehensibly, incompetent, to the point
| where I can only assume they're colluding with Nvidia,
| behind the scenes.
| askl wrote:
| Might be worse because it takes time between designing the
| silicon and having the first usable chips. So they're
| outdated the moment they hit the market or even before that.
| karmasimida wrote:
| A model can't be updated, and a chip that is only relevant for
| 6 months at max?
| askvictor wrote:
| People already buy new phones every year, this just creates
| even more reason to do so
| Gigachad wrote:
| Outside of this website I've never met a person who buys a
| new phone every year. It's closer to every 3-4 years for
| most people.
| boelboel wrote:
| Closer to every 5-6 years these days and with ram prices
| going up it will be even longer. Especially with the
| low/mid range phones, which are most phones outside some
| developed countries, people will keep their phones as
| long as they can.
| Gigachad wrote:
| Would depend on the income levels, but yeah, buying a new
| phone these days is entirely a non essential luxury. An
| iphone easily lasts 7 years so the moment money is tight,
| it's a very easy choice to not buy a new one.
| askl wrote:
| I mean are there even any reasons to buy a new phone?
|
| If I compare the Pixel 6 Pro I'm using at the moment to
| current models, they are functionally identical. The only
| reason to upgrade might be getting a fresh battery and
| access to firmware updates.
|
| Otherwise I'd be happy to continue using it for the next
| 10 years.
| epolanski wrote:
| I only buy phones if the current one starts showing signs
| of deterioration, mostly battery.
|
| I swear a midrange Chinese phone from 2017 would be
| enough for me in 2026 to read HN/Whatsapp and some
| Youtube.
| throwaway240403 wrote:
| Your location/income bias is showing. Most people do not
| buy new phones every year.
| winrid wrote:
| I live in the bay area and buy a phone maybe every 3
| years? Why do people waste so much money :D
| askvictor wrote:
| I never said most people. But it's not uncommon. I
| personally find it ridiculous, and hold onto my phone
| until it's unusable, but plenty of people in middle class
| Australia seem convinced that they need the new one
| whenever it comes out.
| anigbrowl wrote:
| Depends what you mean by relevant. If you use AI primarily as
| a search/knowledge engine, it makes no sense. If it's your
| capable assistant that has a lot of general knowledge, can do
| tool calls, and has a big context window, very doable.
|
| Indeed, for some kinds of applications involving secure/legal
| data etc. I can see the consistency of silicon winning out,
| because it combines performance with immutability and
| guardrails in hardware. Some chips have write-once PROMs to
| store password hashes and similar, you could do the same
| thing with prompt hashing to absolutely force or forbid
| certain behaviors. A model that can't be updated is also a
| model that can't be hacked.
| hamdingers wrote:
| One of these chips smart enough to take orders at a drive-
| thru would be relevant for a decade, minimum.
| simpsond wrote:
| Base model sure, but the stack will be hybrid. It's still
| early days here. Too bad FPGAs have such large feature size.
| mrtksn wrote:
| Isn't that kind of useless for the stock? It sounds
| complicated, unlike having number of CPUs go up.
|
| It's like talking about anything else than Megapixels when
| everyone was convinced that megapixels must go up in certain
| periods of the smartphone boom.
| alightsoul wrote:
| Because Openai and anthropic are not hardware companies. They
| outsource that to Broadcom and AWS' Annapurna labs.
| wmf wrote:
| OpenAI and Anthropic are both designing ASICs.
| alightsoul wrote:
| So they have decided that putting a small LLM on a phone
| would backfire because people would have a negative
| perception of their cloud models. Pretty sure AMD will use
| these taalas chips in data centers, not phones
| giancarlostoro wrote:
| ASICs is what took over Bitcoin mining, cheaper in all ways,
| and lasts longer than Nvidia GPUs for inference.
| SR2Z wrote:
| > cheaper in all ways,
|
| Bitcoin mining doesn't have large memory requirements, but
| does have huge compute requirements. ASICs work great there
| because it's very straightforward to add some circuits for
| computing hashes. If you _also_ have to add many GB of
| memory, then suddenly ASICs will cost as much or more than
| comparable off-the-shelf hardware and they won't be faster
| unless you've also invested in huge memory bandwidth.
| giancarlostoro wrote:
| My understanding is an ASIC can last 10+ years, where are
| Nvidia enterprise GPUs are rated for 5...
| SR2Z wrote:
| Most enterprise GPUs are scrap after 5 years because
| they're so inefficient compared to newer models. It's
| entirely possible to make them last longer by
| undervolting them, people just don't because it doesn't
| make sense.
|
| Bitcoin OTOH has used the same PoW algorithm for a
| decade. Barring some _really_ exciting discoveries about
| the nature of computation, new ASICs are not that much
| more efficient than old ones.
|
| BTC mining is also not exactly competitive anymore; the
| nature of the PoW algorithm means that it's dominated by
| a few large players who've set up shop next to a dam and
| who pay very little for electricity.
|
| New entrants are highly discouraged because the mining
| rewards are constantly halving, it's hard to find cheap
| power, and the price of BTC is now so volatile that a
| yearslong investment is very likely to lose money.
| CircuitSeuss wrote:
| Apparently Anthropic is moving that way:
| https://arstechnica.com/ai/2026/08/anthropic-confirms-plans-...
| mdp2021 wrote:
| Not necessarily: it is relevant to Taalas only if it is a
| compute-in-memory architecture.
|
| The Jalapeno mentioned (<<Anthropic is not alone in walking
| this path>>) in the article is still a classical Von Neumann
| architecture.
|
| And Taalas' idea makes sense in a perspective of scale -
| producing a large number of cards; "for internal use" (a
| lower order of items) means a high production cost.
| throwaway27448 wrote:
| You need to find customers for several-generations-ago models
| before this makes any sense. AMD is a lot more incentivized to
| look than mr vanilla llm is
| la6479 wrote:
| Just to see how fast it is try chatjimmy.ai
| mr_mph wrote:
| Pretty incredible to see. It reminds me of when I first used
| the Groq chatbot, except in this case it's a full response
| instantly.
| tasty_freeze wrote:
| It is really fast and ... really hallucinates. I asked "Does
| the Wang corporation still exist? If not, what happened to
| it?" and it replied (in part):
|
| "Yes, the Wang Corporation, the company that originally
| developed and marketed the Wang 2200 computer, still exists
| as a rebranded company under the name PPL (Precision Pencil
| and Label), but it has undergone significant changes and
| challenges over the years.
|
| Here's a brief overview of what happened:
| Founding and Growth: The Wang Corporation was founded by An
| Wang in 1969."
|
| In fact, Wang labs was founded in 1951. PPL seems to be a
| made up entity. But it did generate those "facts" in 0.033
| seconds. If people value speed over accuracy then I can write
| an LLM that is 100x faster than chatjimmy.ai and make big
| bucks by responding one of N canned responses to any
| question.
| mickaelkerjean wrote:
| their tech is a mere demo to open up a new path, the day we
| can have some asics running a Qwen3.6 27b, this would open
| up new doors
| jjcm wrote:
| to be fair, it's running an 8b model from like 2 years ago.
| Taalas just does the chip design, not the model
| architecture.
| UncleOxidant wrote:
| I guess I'm not understanding why this makes sense for AMD to
| buy Taalas unless they plan to get into hosting. It doesn't
| seem like a great fit.
| stingraycharles wrote:
| Didn't Anthropic acquire Cerebras? Seems like a move into the
| same direction.
|
| I also think that etching models into ASICs may be a bit too
| inflexible for what OpenAI and Anthropic want.
| wyrdcurt wrote:
| No, that's backwards. OpenAI are the ones investing in
| Cerebras. Part of the deal is that they can't sell to
| Anthropic.
| wraptile wrote:
| This seems like a very bad and dangerous direction for our
| society.
| unsigner wrote:
| Their thing is improving the models; it would be extremely
| counter-company-culture to bet on models plateau-ing. Maybe
| wise in terms of hedging, but still difficult to pull of as a
| company decision.
| Haven880 wrote:
| Chinese already start making DUV which can do the lower end
| 7nm. They are winning. Once that 7nm and up market cornered by
| Chinese, AMD Intel and TSMC and Samsung will have to burn thru
| bleeding edge depreciation faster perhaps from 7yr down to just
| 18mths. The CPU they generated will be incredibly expensive.
| Meanwhile Chinese just keep minting the AI cheaply and more
| efficiently and inching upwards towards 1.4nm.
| petra wrote:
| They can do 7nm. But they use multiple patterning(printing
| the same pattern multiple times to get to 7nm), which is
| expensive. So it's not comparable on cost to western single-
| patterning 7nm, and of course not to the leading edge on
| cost/size/power.
| planb wrote:
| They are: https://openai.com/index/cerebras-partnership/ My
| guess is they only consider Luna "good enough" to justify the
| immense up-front investment to put it onto silicon, but Luna at
| 10x the current speed would be killer. If they're really
| pursuing live voice conversations with a hardware assistant,
| latency is more important than accuracy (for complex questions
| the assistant could always say something like "wait a minute, I
| need to think about this" and hand over to another model).
| ldng wrote:
| How ? Do LLMs actually "know' when they don't "know" ?
| Certhas wrote:
| How do humans?
| dgellow wrote:
| Always the same trick of not answering the question and
| deflecting to ,,what about humans". Can you folks not
| evaluate LLMs as the system they are, without vague
| gestures at how a different system behaves?
| Certhas wrote:
| Evaluating LLMs is incredibly difficult. They are
| categorically different from any other system we have
| intuition about.
|
| That said, I read the question I am replying to as a
| rhetorical one. If it was meant as a genuine question,
| curious about the question of meta knowledge, then I
| misread. Certainly the question is extremely interesting,
| for both LLMs and humans! But it's also obviously a very
| difficult one, as we don't even have a clear theory on
| how "knowing" works in the base case.
| planb wrote:
| They do this all the time, I'm using ChatGPT in Instant
| mode and it auto updates to thinking if my question is
| complex. Most of the time this works.
|
| To answer your question: A large language model itself does
| not know this (afaik). But chatbots are not "just LLMs" but
| a whole bunch of systems (and models) around them.
| ldng wrote:
| Ok, but the article is about etching the model, not a
| "whole bunch of systems". So far, I still don't if its
| actually doable or if it is just unsubstantiated
| speculation.
| Philip-J-Fry wrote:
| Cerebras doesn't etch the model onto silicon though. They're
| basically just wafer scale GPUs. They're more flexible than
| etched silicon though because they can just run the next
| version of the model almost straight away.
| planb wrote:
| Interesting. I did not know that. And still they got 15x
| speedup (https://www.cerebras.ai/blog/openai-gpt-
| oss-120b-runs-fastes...). I wonder how much additional
| speedup would be possible by really etching the model.
| larodi wrote:
| We don't really known (from the outside) how exactly do they
| move. Besides it may have not been truly viable 1-2 years
| ago...
| elAhmo wrote:
| They were busy buying open source frameworks and teams behind
| those.
| bjackman wrote:
| Dwarkesh recently pointed out [0] that these guys are almost
| forced to spend most of their compute on training instead of
| inference. This is because they need to maintain the appearance
| (which may also be the truth) that future models will make
| current models obsolete and be much more valuable.
|
| Completely fixed-function HW can't be used for training, it's
| inherently a statement that "this model is Good Enough and we
| are now gonna start just extracting its value instead of
| extending it". So yeah it's an inference moat but it's not a
| growth moat.
|
| Makes perfect sense for a company trying to get into the
| compute business, not companies who wanna be in the creating-
| ASI business.
|
| Still, I guess/hope they have teams doing it in-house anyway.
| Just not something they'd wanna make a huge amount of noise
| about, it doesn't look good for To The Moon valuations.
|
| [0] https://www.dwarkesh.com/p/why-compute-might-get-10x-more-
| ex...
| troyvit wrote:
| I wish we lived in a reality where Framework was anywhere near
| rich enough to acquire them. I'd love to have models on a chip
| that I could swap in at a whim. That would do the opposite by
| eliminating moats.
|
| I guess there's a tiny chance AMD makes something like that
| happen. It seems like a great way to get people and orgs to pay
| a few hundred bucks every 6 months or so.
| vonneumannstan wrote:
| >Google is already doing this and has an experimental project
| on top of already having TPUs and cramming their quantized
| flash onto individual TPUs for inference.
|
| Google is no longer a serious player in frontier AI. I doubt
| they will ever hit a SOTA model again.
| adityazero wrote:
| They are busy capturing market first, and I think that makes
| more sense. 'premature optimization etc.'
| walrus01 wrote:
| Imagine the size of chip needed to 'etch' something like Qwen 3.6
| 27B in size.
| flog wrote:
| If someone has that sort of knowledge; how big a chip would be
| required? Is it possible?
| mdp2021 wrote:
| Well, given the data above, roughly a 220b transistors chip
| for the HC1 tech.
| mdp2021 wrote:
| Not too dissimilar to the first HC1 (6nm 815mm2 53B Transistors
| embedding an 8b LLM):
|
| > _Our second model, still based on Taalas' first-generation
| silicon platform (HC1), will be a mid-sized reasoning LLM_
| golem14 wrote:
| Interesting thought, because it's a yield question. How
| tolerant are models today to a few broken weights.
|
| If tolerant, they could churn out many cheaper chips, some
| perhaps with slight abnormal tendencies ;)
| walrus01 wrote:
| I wonder if you had a few percent of problems in the yield,
| if it would be functionally equivalent to the difference
| between a unsloth-published Q6 standard size GGUF vs. the
| nearly perfect precision of an unsloth Q8-K-XL. Or more like
| Q4 vs Q8 where a lot is lost.
| thepasch wrote:
| > How tolerant are models today to a few broken weights.
|
| Extremely! You can remove entire _layers_ and the model will
| still work just fine, with barely perceptible capability
| losses.
|
| I've cut/bypassed ~15% of total parameters out of Gemma 4 31B
| on a pod once. Still got perfectly coherent responses out of
| it. Certain layers are a lot more important than others,
| particularly early and late ones; but it's honestly
| astonishing how much can be cut out from the middle without
| destroying the model's coherence.
|
| I didn't run any meaningful benchmarks, so I have no idea
| what the capability loss looks like exactly. But "produce
| coherent and sensible English in response to a wide variety
| of prompts" was definitely not among the things the model
| unlearned.
| walrus01 wrote:
| Brings to mind the scene in '2001' where Bowman is pulling
| out individual pieces of hardware that represent the mind
| of HAL, and it becomes increasingly incoherent as more
| physical hardware is detached.
|
| https://www.youtube.com/watch?v=UwCFY6pmaYY
| nojs wrote:
| Can anyone comment on the economics and likely turnaround times
| of this process, when it's more mature?
|
| Would it be realistic for a frontier lab to deploy this or would
| the turnaround time mean the model is always too out of date?
|
| Assuming the weights and architecture are eventually stable, how
| much cheaper would this end up being?
| 2001zhaozhao wrote:
| There are always uses for outdated models.
|
| Claude Code is still using haiku 4.5 from ages ago for explore
| subagents for instance. Not to mention production uses like
| customer service that only need to be "good enough"
| edot wrote:
| Just looked this up, no longer true. Explore subagents
| inherit whatever model the parent is. And you can of course
| make other subagent configs.
| samtheprogram wrote:
| That's solely so that you burn more money. It's totally
| unnecessary to assume the parent model. Sure, it could be
| upgraded from Haiku if there was a solid reason to, but...
| AussieWog93 wrote:
| I mean, if you could get Opus or even Sonnet 4.5 at 1000+
| tok/s exploring the codebase, they would probably change
| that setting back.
|
| But either way, I think GP's overall sentiment of
| "delegating intelligence-saturated tasks to an outdated but
| fast subagent" makes a lot of sense.
| alightsoul wrote:
| Customer service has really degraded huh. 4 years ago they
| expected opus performance out of human call center agents
|
| I guess losing some customers due to poor customer service is
| ok if the price of customer service is right.
| shangofox wrote:
| I mean even if it take a few months, it'll still be out of
| date. But there was a hypothetical when it came up in Feb,
| would you want Qwen 3.5 at like 10k tokens per second.
|
| At the time people were no doubt saying yes but now 3.8 is out,
| is that still desirable?
| xienze wrote:
| There's soooo much stuff that such a model is still capable
| of doing in the pursuit of getting a better overall answer.
| Imagine a powerful research agent that blasts out dozens of
| the small, cheap models to fetch and summarize one page each.
| Then the beefy researcher model performs the final analysis.
| cogman10 wrote:
| 2 to 3 months optimistically assuming everything goes smoothly
| and is fully automated.
|
| 6 months or even a year if something goes wrong in the
| fabrication process and you need to update things.
|
| If they do more standard asic design, it could be a lot longer
| as the design needs to be validated on an FPGA cluster, which
| would necessarily need to be very big for something like a LLM.
| Easily up to 2 years.
|
| There's a reason chatjimmy isn't demonstrating newer models and
| why they only show of an 8B model.
| andrewvl wrote:
| It must be a "super model". What will be if new model released?
| New chips?
| downrightmike wrote:
| Chip pops out like a gameboy cartridge. AI not working? Blow on
| it and jam it back in
| andrewvl wrote:
| Agree. Price is the question.
| tecoholic wrote:
| With web search and tool call a decent current generation model
| at the speed of the chatjimmy could do a lot. People saying it
| would be out of date are missing the point. It's not going to
| make much sense for frontier companies that's chasing the SOTA.
| But for a lot of business use cases if someone can put GLM 5.2
| and sell it as a box, it would make so much sense.
|
| My partner has been asking for a "completely private" model for
| doing research and shifting through volumes of data that can't
| leave the office and $$$ for the current hardware makes no sense.
| It would be an easy sell if someone walks in with a black box
| that contains "ChatGPT".
| equinumerous wrote:
| 100% agree - you don't need the most up-to-date model to have
| something that's useful in agentic contexts. They could even
| produce chips with weights that make all the decision
| making/logical reasoning and have it delegate to other
| specialized agents. If it becomes cheap enough to print a run
| of custom chips, releasing a batch for each major advancement
| does not seem unreasonable for SOTA companies.
| cephei wrote:
| There are so many use cases for supremely fast offline models.
| The first thing that comes to my mind is for real-time video
| processing or other non-textual content in real time.
| anigbrowl wrote:
| I wouldn't call it _supremely_ fast but zippy and versatile,
| yes: https://shop.m5stack.com/products/ai-pyramid-computing-
| box-p...
| 5555watch wrote:
| In my understanding the first Deep Think / Pro models were
| already very good as they were doing some kind of parallel
| repeated reasoning, thus were slow and expensive. So if
| chatjimmy speeds enables a fast deep think level performance, I
| think that would be great.
| yumraj wrote:
| Given the fast churn of the models, how does it work out?
|
| Won't the silicon etched model already be 1 or more versions
| behind by the time the silicon comes out.
|
| Though if it's cheap enough, there certainly can be a market for
| cheaper model inferences.
| sigmoid10 wrote:
| I find speed alone would be a game changer for current models.
| I hardly find any task anymore that the current frontier models
| can't do with max reasoning after several rounds of feedback
| (provided sufficient instruction and the right harness). But
| waiting an hour or more for reasoning to finish is getting
| really cumbersome. If they could do the same in seconds (and
| for cheap of course), I'm pretty sure we'd pretty soon see
| major software companies pop up that are run by a single human.
| deadbabe wrote:
| Can you give some examples of these tasks that require an
| hour or more of reasoning?
| xyzsparetimexyz wrote:
| The recent maths prompts did. The 'you should find a
| breakthrough' one was several blocks of reasoning, each
| taking 90 minutes or so
| sigmoid10 wrote:
| I've let it research and implement pretty complex stuff for
| my 3d engine. Like a realtime volumetric cloud rendering
| solution. The kind of stuff you can do yourself, but it
| would probably take weeks, if not months. I spent quite a
| bit discussing the design with it, knowing some details of
| how it should work from game dev conference talks. Once we
| agreed on a design and implementation plan with key steps
| and intermediaries, i just said "now implement it" and it
| happily designed algorithms, wrote and ran tests, generated
| and looked at screenshots and basically did everything you
| would expect from a really senior engine programmer for
| more than an hour of wall time. The end result was
| something that worked amazingly well after testing it
| myself and suggesting minor corrections that were my fault
| for not specifying them in the first place.
| prinny_ wrote:
| They expect a sort of breakpoint at which each subsequent model
| version will only be marginally better than the previous ones,
| thus allowing them to retain their value for some time. Their
| business doesn't work if each year the new model demolishes the
| previous one in terms of performance.
| deadbabe wrote:
| You could take your silicon chip and have it re-etched only
| with model diffs for an upgraded version.
| yumraj wrote:
| How does that work, as in re-etching of silicon? Any pointers
| to read?
| deadbabe wrote:
| Someone will figure it out.
| XCSme wrote:
| I think this would make sense for consumer hardware, not for AI
| companies.
|
| AI companies constantly update/change stuff, new models come
| out, new requirements, etc.
|
| But if you ship an "ai-powered" dishwasher, it can come with
| the chip built-in to do computer vision and precisely target
| each spot, and will be sold as-is with no updates.
| m463 wrote:
| subscription "ai-powered" dishwasher with personalized user
| ads, most of the chip dedicated to "personalized" not spots.
| XCSme wrote:
| So local personalized ads?
|
| Not sure if that's better or worse than online
| personalizaed ads...
| throwaway173738 wrote:
| You don't need this chip to do that. Computer vision has used
| machine learning for decades. The task you're describing is
| pretty rudimentary and an off the shelf model with a control
| system would do it way cheaper.
| XCSme wrote:
| It was just a random example, you could think of it as
| being a lot more complex (detect which type of food it is,
| what detergent to use, how much water, remember patterns,
| learn over time, adapt, etc.)
| tyre wrote:
| Think of a HomePod. 99% (and likely much more) of what
| people are asking is super simple.
| yumraj wrote:
| Makes sense. Actually to expand, I believe this can make a
| lot of sense for industrial robots and such which have a more
| or less fixed job and latency matters more, so a well tested
| model may be more valuable than need to keep updating them
| thunderbird120 wrote:
| It makes fairly little sense for consumer hardware, unless
| you're embedding an absolutely tiny model. Since you're
| embedding the entire model in the chip itself, required chip
| size scales directly with model size. The reason they have
| llama3.1-8B as their model in https://chatjimmy.ai/ (their
| demo) is because it's very hard to do anything bigger than
| that, you literally start running out of physical space on
| the chip, and bigger chips cost money. You can't have a
| dishwasher using a chip the size of a B200 die to run a model
| you could run on a Raspberry Pi.
|
| What's worse is that this is when it's already quantized to
| ~3 bits per parameter (which is fairly lobotomized). Yes, the
| chip will run it 1000X faster than the Raspberry Pi, but it
| will only be stupid faster.
|
| Their press release explicitly lists that their HC1 puts the
| entire Llama 3.1 8B model on one 815 mm2 TSMC N6 die, with
| about 53 billion transistors.
|
| 815 mm2 is fucking huge. An RTX 5090 is 750 mm2. A mid-to-
| high end consumer CPU die is something like 300 mm2, with a
| lot of budget parts being significantly smaller than that,
| down to ~70 mm2. Every square mm costs money.
|
| If they upgraded from N6 to N3P they MIGHT be able to get as
| much as a ~35B class model on a a chip which fits in the
| reticle limit, maybe, probably not, because model weights
| aren't the only thing that needs to fit on the chip.
|
| There are very serious issues with agentic performance in
| this setup, which is exactly where you would want something
| really fast. Their Llama 3.1 demo lists a context of 6,144,
| which is dramatically lower than the 131,072 Llama-3.1
| supports.
|
| Reasoning models are barely usable with contexts that short.
|
| The reason for this is that to actually get those speeds, the
| KV cache needs to live in SRAM. You can't bake the KV cache
| into the circuitry since it... changes. They clearly don't
| have enough SRAM, and the problem gets worse the bigger you
| make the model since KV cache grows (sort of) with model dim.
| The longer you want to make your context, the more of your
| chip needs to be SRAM.
|
| Frankly, I don't see the use-case for this tech. It's too
| expensive and too inflexible. Just doing what Cerebras did
| and making a wafer-sized chip which is mostly SRAM is a much
| better solution to serving LLMs at extreme speeds and you
| don't need to make a new chip every time a new model comes
| out.
| hypfer wrote:
| And yield. Good luck getting good yield out of such
| ginormous chips.
|
| That said, I've been wondering if they could go with
| multiple smaller ones instead. Like one per layer maybe
| even?
|
| What are your thoughts on that? You seem to be more
| qualified than me on that matter.
| thunderbird120 wrote:
| Lots of chiplet interconnects hurt your max performance
| pretty severely and drives up your complexity a lot.
| There's a reason they want to keep it monolithic.
|
| I already harped on Cerebras, but their approach of just
| copy/pasting a whole bunch of identical functional
| blocks, over-provisioning the chip by ~8%, and then just
| fusing off blocks with defects allows them to effectively
| have 100% yield on a wafer-sized monolithic chip. This is
| very desirable, and just another reason I like their
| approach better.
| XCSme wrote:
| Yeah, makes sense, if it's good for very small models only,
| then there's no point, as those van already run on cheap
| consumer hardware.
|
| Yet, maybe it can work well enough, so that as a
| manufacturer, you don't pay $50 for a PI, but only $0.50
| for a tiny "hard-coded" chip.
|
| The advantage can be that, as a LLM, as opposed to other
| types of chips, the use-cases could be more varied, so same
| chip could be use in different devices (robo vacuums,
| security cameras, ball-shooting training robots, etc.)
| petra wrote:
| Maybe AMD will do a 3D memory layer for this chip for the
| kv cache. Possibly even something more dense than SRAM,
| like cgram, when it's ready.
|
| Could this enable a reasonable context size ?
| laweijfmvo wrote:
| pretty much everything is "1 or more versions behind" by the
| time it comes out. the question is whether or not it's still
| useful? at some point, presumably not every application will
| need the latest cutting edge huge model.
| brokencode wrote:
| Already models have gotten really good at a lot of things.
|
| A lot of people would probably be happy to stick with the same
| model for a year or two if it's 10x faster and cheaper.
|
| And perhaps older models can become cheaper over time as newer
| models come out on new silicon for a higher price. That
| incentivizes people to stick with older models.
| christina97 wrote:
| There's some kind of tradeoff between speed, cost, and quality
| for every application. I would be perfectly happy with a model
| 6 months old that was 50x faster for many uses. Right now I use
| either Opus (for smart stuff) or Flash without thinking (for
| fast stuff). I would take an even dumber model for more speed
| (lower latency in particular).
| craftkiller wrote:
| I think the real value here is not as a customer-facing
| agent/chatbot but for for automated processes. Think of all the
| companies out there that have LLMs doing simple tasks like
| categorizing customer feedback emails. For such tasks, you
| don't gain much from better models, so if you could run it 10x
| cheaper on a slightly older model, it would absolutely be worth
| it. Pretty much any place people are currently running a flash
| model could benefit from this since they're already deciding
| that speed+price is worth using a less capable model.
| Maxion wrote:
| Or as subagents reading code or other text and summarizing
| them.
| nullbio wrote:
| Perfect for consumers. You buy it and then you need to buy a
| new one in a couple of years. If they can make them affordable
| they'll sell like hotcakes.
| etoxin wrote:
| And the second hand market. I'd love to see this integrated
| into motherboards like RAM. Someone could have a motherboard
| with 4 sticks of different AI with various models. Swap,
| change and trade.
| chorizo wrote:
| That's not going to be true forever. As models mature, we will
| hit diminishing returns. Major improvements will come annually
| rather monthly - matching the roughly annual release of new
| processors. Model ROM's will likely get integrated into die
| packages just like DRAM now.
| pennomi wrote:
| I'm hoping for SNES style cartridges
| mrheosuper wrote:
| I'm still using Opus for most daily task because Fable is too
| expensive.
|
| If they begin etching Fable into silicon now and release it 2-3
| years later, i can see the market for it
| noosphr wrote:
| This is a feature for most local use cases. You don't want all
| your work flows to start failing because of a model update.
| casey2 wrote:
| There isn't a fast churn in the underlying pretrained model,
| nor RL. It's mostly orchestration around the model. Said
| another way you could just pretrain and RL for longer.
|
| Also I believe there is both a market for extremely fast local
| inference with current model performance and that such fast
| inference would unlock unforeseen usecases. Especially as TPS
| approaches early computer clock cycles and data rates.
| hahahaa wrote:
| They still make 6502s right.
| cdolan wrote:
| We have some processes running on models released a year ago
| (which we're updating, but still)
|
| The speed is incredible. It doesnt matter if you are ~30-300
| days behind
| trklausss wrote:
| The thing is, right now it is exploding because we are at the
| beginning of it. At some point, it will plateau at a specific
| level, and not that much quality will be gained. There is
| however leaps to make for efficiency.
|
| The same can be said about the CISC computer: yes, new
| processors introduce new instructions that do something
| slightly faster, you could still crunch that with an older
| processor. The real benefit comes in clock cycles (that's why
| Arm with a reduced set can compete with x86).
|
| Also: there are myriads of models, for myriads of tasks. Not
| all have the same development gains as we see for general
| purpose AI. If you etch those, you reduce your bill by factors
| down.
|
| It also democratises models: Instead of running them on a cloud
| server by some company, you can run them at home, for coding
| tasks, without the need of internet connection, etc.
| woodrowbarlow wrote:
| also, as this scales, what would this mean for closed-weight
| hosted models? i imagine it's possible (but difficult) to re-
| derive model weights by de-lidding and inspecting the die... so
| will this only ever be used for open-weights models?
| msteffen wrote:
| This is neat but IMO a little crazy.
|
| Something I personally haven't seen much of, in all the
| discussions of model benchmarks and AI breakthroughs, is a
| distinction between "peak performance" and "reliable
| performance". The "peak performance" of frontier models is very
| high: they're solving open math problems, analyzing large
| codebases, etc. But my subjective impression is that "reliable
| performance" is mid at best: out of 100 random questions I might
| think to ask, it's likely to say something wrong or stupid a
| handful of times at least.
|
| I think there's inherent tension between the two: the more a
| model reaches or outright hallucinates, the more likely it is to
| come up with tricky, subtle solutions to problems (I think people
| are somewhat like this too: Terry Tao's brother is nonverbal, Jim
| Watson's son has severe schizophrenia, etc). But then the less
| likely it is to generate a sensible email reply.
|
| I use models all the time for coding, but I would not let one
| take over my daily correspondence. If the idea here is to run
| frontier models at high speed in data centers, that could be
| useful (the speed would be cool), but I'd be surprised if the
| cost of that hardware churn is worth it to frontier labs. But if
| the idea is to turn this into a chip that goes in your phone as
| some kind of routine, low-power inference thing...taking
| something too kooky to be relied on and baking it into your
| phone's hardware like that doesn't make sense to me.
| daishi55 wrote:
| > out of 100 random questions I might think to ask, it's likely
| to say something wrong or stupid a handful of times at least.
|
| What are some examples?
| wmf wrote:
| There's a benchmark for this and a lot of models get negative
| scores because they're so unreliable:
| https://artificialanalysis.ai/evaluations/omniscience
| daishi55 wrote:
| I wanted some examples they actually experienced. Because I
| use these things daily and haven't seen a hallucination in
| a long long time.
| yoyohello13 wrote:
| I saw a hallucination just this afternoon about a
| spurious ca cert error. Definitely happens less often,
| but I do need to correct it occasionally. Maybe once a
| week so it still requires vigilance.
| tyre wrote:
| Search a terminal with Claude Code for things like, "I
| got it wrong twice. I should look up the documentation
| instead of guessing."
|
| Does it about once a day, that I notice.
| B-Con wrote:
| Not OP but I've tried using all the major frontier models
| to find niche items in a field that I collect. The models
| get confused and hallucinate items that sounds complete
| reasonable, but don't actually exist. Usually it's an
| amalgamation of several real products.
|
| My prompt is akin to "recommend <item type> with <niche
| criteria>". The first 3-ish results are about right, and
| then 7 of the next 10 are hallucinations and the LLM
| clearly can't throw up its hands and say "I got nothing".
|
| I'm sure this is a hard problem because of a) how many
| items there are, b) how much overlap there is between
| product names, descriptions, manufacturers, different
| versions of the same product, etc, so keeping them
| distinct in the model's memory is probably hard, and even
| worse if it is dynamically fetching and summarizing
| content then it will be very easy to conflate different
| items, and c) LLMs are known for not working well on the
| edge cases with few examples.
| msteffen wrote:
| Well, I wrote my comment after an exchange similar to this,
| with the linked example chatbot (chatjimmy):
| https://pastebin.com/uEV7Pm8q
|
| I asked it a slightly tricky math problem (I re-asked it the
| same problem to create the paste, and it did about as well
| the second time). It was unable to solve the problem, and
| it's a small, old model, so...fair enough...but also its
| answer was pretty incoherent, with stuff like "Since A is an
| invariant set, it's always possible to find a cave that the
| fox cannot be in. Therefore, you can always catch the fox in
| that cave." (...catch it...in the cave it can't be in?).
|
| Then, off the top of my head: Claude somewhat recently
| generated a Spark Job where the worker timeout was longer
| than the worker heartbeat, so workers would always inevitably
| be killed when they didn't heartbeat within their timeout
| window. (also...neither option needed to be set?) Before I
| noticed the problem, I asked Claude why the job was taking so
| long, and it told me the data set was too large. More
| recently, there was a blog post by John Scalzi I was having a
| hard time finding, so I posed the problem to ChatGPT, and it
| came back with a blog post that didn't include any version of
| the text I remembered and wasn't really topically relevant
| (and maybe _I_ hallucinated the blog post, but it could've
| said "I can't find it" instead of "here you go"). On another
| occasion, I was trying to find a particular episode of Bob
| the Builder for my kids, so I Googled it, and Gemini kept
| giving me the wrong season and episode number, even after
| several rounds of "no, s5e6 is 'that thing', I'm looking for
| 'this thing'." Turned out the episode wasn't on Amazon at all
| (which _I_ had to tell _it_ ), and I had to go find it on
| YouTube.
|
| That said, as I sit here scrolling through my history to see
| if I've forgotten any particularly good examples, I have to
| admit they do a better job than I'm giving them credit for.
| But I still wouldn't have them write my email for me (the one
| time I tried that, when I was playing with openclaw, it sent
| a fairly demanding email to someone I didn't know that well
| without asking for confirmation, and I had to go apologize
| and explain that I hadn't really written the email, which was
| embarrassing), nor am I particularly excited to have
| chatjimmy as a permanent resident of my pocket.
| dumberquestions wrote:
| I think you're underestimating both their reliability for
| standard problems and the usefulness of that level of
| reliability.
| tyre wrote:
| This is a good point. Opus does some silly shenanigans
| sometimes but then catches it later. It's still an order of
| magnitude faster at getting to a working system than I am,
| for ones I don't know.
|
| It's really a dream for setting up a homelab
| jgilias wrote:
| What I'm doing is running a kind of a meta harness that uses
| different models (and underlying harnesses) to work on a
| problem, or review the solution. Idea is to get to an error
| rate better than each of the underlying models can provide.
| Same thing as sensor fusion.
|
| Now, that's slow and expensive although seems to work quite
| well (haven't really evaled this properly, don't have the
| time). If inference can be made fast and cheap, multi-model
| approaches like this would become more viable for more
| applications.
| Tade0 wrote:
| Up to a certain scale a model only needs to effectively do tool
| calls - that is something which doesn't require too frequent
| updates.
|
| What I imagine an on-device model should be doing is just
| translate natural language to search requests and calls to
| tools manipulating retrieved data - much like no model
| currently does calculations and instead they open up calculator
| and use that instead.
| cubefox wrote:
| > At 20 billion parameters per chip, you'd need just 50
| accelerators to support a trillion-parameter model
|
| I don't see any evidence that this is possible. From my
| understanding, the whole model needs to be on a single chip.
| Which rules out any popular frontier models with several
| trillions of parameters. Even smaller sub-frontier models have
| hundreds of millions of parameters, so these would be ruled out
| as well.
| IsTom wrote:
| I think it's enough that a single layer fits on each chip if
| you can daisy-chain them with good interconnects.
| wmf wrote:
| The methods for splitting weights across multiple chips are
| well established. Groq/Cerebras can't hold a model on one chip
| either.
| pyrolistical wrote:
| Umm I have an extra 35, do you have layer 6?
| octoberfranklin wrote:
| They pipeline-parallelize across multiple chips. DeepSeek v4
| Pro will be 30 chips.
| redmoonx wrote:
| It obviously won't be continuous delivery but could make sense if
| the lifecycle of a model (train, deploy, iterate (meaningfully)
| is about 1-2 years. In that case it fits nicely in the "this
| year's model" already established with cars, phones, etc.
| ggm wrote:
| Field reprogrammable, it's an FPGA on steroids. Field upgradable.
|
| Burnt in, it needs a zif socket and easy access in every car,
| aircraft, a pull out slot in a phone, or it's new era planned
| obselescence.
| mdp2021 wrote:
| Can that be done when the whole idea is to store a multiplier
| into a handful of transistors?
| ggm wrote:
| I have no idea. It makes my comment a statement posted as a
| proxy for a question, a question you correctly pose
| explicitly.
|
| If it can, then deployment in a sea of gates can make a chip
| viable across model generations as weights change, inside
| some scale factor.
|
| If not, unless the part is under a pinout and address model
| which can scale on the bus, and can be easily replaced, it
| makes the entire dependency a replacement, not just this
| part. So embedded use has consequences.
| XCSme wrote:
| Why not have some a device/hardware that programs itself on-
| boot.
|
| Sort of a FPGA, that (electrically) arranges the connections
| on-boot, and then it's like a static inference chip.
| wmf wrote:
| FPGAs already configure themselves on boot.
| XCSme wrote:
| I asked a LLM after posting my comment, to see if I had a
| genius idea or not,just for it to tell me the same as you,
| that's now they work already...
| xyzsparetimexyz wrote:
| It can just be pcie
| redox99 wrote:
| Is there any LLM from exactly one year ago that would be worth
| running?
|
| In Aug 2025 you had
|
| - OpenAI o3
|
| - Opus 4.1
|
| - Gemini 2.5 Pro
|
| - Grok 4
|
| Even if those were almost free to run, you'd be way better off
| with Deepseek flash 0731 or GPT 5.6 Luna, which already are
| almost free.
|
| Other than for things where the t/s are critical, it seems like a
| bad idea to etch a model into silicon.
| daishi55 wrote:
| That is fkin wild. o3 was just a year ago? The progress is
| truly insane.
| redox99 wrote:
| Yeah I had to double check, o3 feels like it was ages ago.
| But GPT 5 came out Aug 7, so it's only one day off from my 1
| year ago cutoff!
| mdp2021 wrote:
| > _Is there any LLM from exactly one year ago that would be
| worth running?_
|
| Bad perspective: consider the correction: "when are thresholds
| of sought quality reached"? Hence: not "is there a 10yo from
| last year that could compete with the current 13yo", but "will
| there be a 30(?)yo from last year that could compete with the
| current 33(?)yo" ('(?)': the scale of yearly growth in the
| future is uncertain).
| redox99 wrote:
| It's not just about it "being smart enough". It's about there
| being actual user demand when it needs to compete with the
| shiny new model.
|
| A 10 year old iPhone is probably good enough, but is there
| demand for it? In a vacuum a 10 year old iPhone is good, but
| why would you pick it if you can have a current one for a
| reasonable price?
| singingtoday wrote:
| We still run GPT 4.1 for some of our use cases. We want to
| replace it but are having trouble finding models that are as
| fast with similar or better intelligence.
| redox99 wrote:
| There's nothing fast about GPT 4.1. It's ~50 tps AFAIK. Of
| course it doesn't use reasoning, but you can run modern
| models without thinking as well. GPT 5.6 Sol without
| reasoning should destroy it in intelligence.
| itgoon wrote:
| Gemini 2.5 Pro was very good at writing single, somewhat
| complex functions. Sure, the rest of the loop would still take
| time, but nearly-instant implementation? Sign me up.
| anentropic wrote:
| What if it was Fable 5 baked in?
| jackdoe wrote:
| Can you imagine in few years getting Fable level intelligence at
| 20k tokens per second?
|
| "You are not prepared" --Illidan Stormrage
| preommr wrote:
| People are missing the point if they think this is useless
| because frontier models keep changing every few months.
|
| We really, really need better secondary models that can do things
| fast and do them cheaply for lots of dumb tasks. Not only because
| it can be used as sub agents by frontier models, but also because
| it can be like a universal grease for all kinds of software.
|
| I've got an app I am building and I don't want to tie myself with
| frontier models because I'll never be able to beat
| openai/anthropic. I just want a simple, cheap, instantaneous
| model that can just go through my documentation and tell the user
| what to do next and how to integrate with whatever ai
| subscription they have.
| ur-whale wrote:
| Yeah, so https://chatjimmy.ai/ ... the model is crap, but the
| speed is amazing. Worth checking out.
| empiricus wrote:
| Worth wondering why they used a crap model.
| jauntywundrkind wrote:
| Core rope memory is back baby!
|
| Enjoying the Ian Cutress / TechTechPotato video on Taalas. Some
| ok good technical details on the tech, and some good insider
| baseball, whose who stuff. (What a treasure having tech
| discussions like this about.) https://youtu.be/3MKRjt59hh4
| laweijfmvo wrote:
| I've been using Gemma as my default (via Kagi) because it's
| served on Cerebas hardware. The speed is honestly a game changer
| for day to day queries.
| drob518 wrote:
| So, Kimi K3 in silicon sometime soon?
| api wrote:
| I've had an endgame idea in mind for a while.
|
| Models, probably first open weight ones like Kimi K3 class, are
| etched into silicon like this and sold as cartridges almost like
| old school game cartridges.
|
| You buy a USB-C dongle that the cartridge goes into, or for data
| centers you have PCI cards that take these in slots.
| wmf wrote:
| Each cartridge costs $1,000. Do you still want it?
| api wrote:
| Me? Probably not. A business or a hoster, sure. There'd
| probably end up being an aftermarket in used cartridges with
| slightly older but still good models on them.
| anigbrowl wrote:
| For fast Kimi K3? You're damn right I do
| wmf wrote:
| $1,000 only gets you the Qwen 27B cartridge. For Kimi K3 it
| would be more like $100,000 (and the "cartridge" is the
| size of a refrigerator).
| trollbridge wrote:
| I would gladly pay $100,000 for local K3 running at
| 18,000 tok/sec.
| anigbrowl wrote:
| Some hurried goalpost-moving there! If we take Qwen 27B
| as the benchmark of equivalent power (rather than its
| default configuration with no fine-tuning), I can still
| think of applications. Auto summarization for legal
| document databases and other high security/offline
| contexts, contraband detection, moderating websites like
| HN etc :)
| singingtoday wrote:
| Yeah. I have 3 max20 plans.
| roughly wrote:
| How's that jive with the fact that they're introducing a new
| model every other week?
| drchickensalad wrote:
| The new model every week is not necessary at this point really.
| What if you could run opus 5 for the next couple years at 1/20
| the cost?
| roughly wrote:
| What's interesting about this is that I as a user would find
| this useful, but I think the AI industry as a whole would
| find it an absolute goddamn disaster. Opus 5 is a very good
| tool, but it is not a human-replacement-level intelligence,
| which means the entire revenue stream the industry's built on
| - labor replacement - is not met by this, and the only
| slightly charitable read of the industry's finances is that
| they're gonna bootstrap their way to creating the labor
| replacement hypothesis by getting people to spend money on
| Opus/etc, whereas if the actual product is a 1/20th the cost
| Opus-on-a-chip, the entire business and financing model
| that's tying up $N Trillion dollars of investment money goes
| out the window.
|
| Great for us, looks like a recession as far as the Market is
| concerned.
| jaggederest wrote:
| Pipeline the burn into silicon, lower the latency as much as
| you can, for the 10-100x operation cost it's worth it. Imagine
| if frontier models cost $5/mtok and the 2nd or 3rd tier models
| cost $5/billion tokens for 3-month-old models.
| moralestapia wrote:
| Taalas is just a phenomenal startup from Toronto. My dearest
| congratulations to the founders.
|
| Edit: Lol, downvotes? Stay jelly, meanwhile Talas goes brrr.
| yousif_123123 wrote:
| If things like this get traction, will we need all the
| datacenters?
| downrightmike wrote:
| You are mistaken about what the datacenters are for
| tripledry wrote:
| What are they for?
| downrightmike wrote:
| open air prison
| mNovak wrote:
| What I like about this, is that it significantly increases the
| probability of a sci-fi scenario where you're picking up a hot
| chip on the black market; rumor has it, Mythos 9 weights baked
| in...
| bigyabai wrote:
| Plug it in, and it's a old prototype with Gemma 5 weights baked
| onboard. Dammit, fucked by Craigslist _again!_
| NitpickLawyer wrote:
| Back in the kazaa and limewire days, you'd sometimes try to
| get a movie / episode from a series, wait hours / days for it
| to download, and when it was done you had a ~50/50 chance to
| actually watch what you wanted or an old german porn movie :/
| rullelito wrote:
| Better then vice versa!
| matheusmoreira wrote:
| Black market uncensored heretic Mythos weights...
| arlort wrote:
| steadily heading towards the final goal of computers being
| composed of pluggable crystals as stargate intended
| pragma_x wrote:
| What's old is new again. We'll be buying cartridges for AI
| capabilities on future architectures.
| sgc wrote:
| What does it take to go from here to a model on a pcie card or an
| m.2 card, so I can plug one into my workstation / laptop? Will
| 'intelligence' become much like a gpu, where most people just
| live with the performance of whatever they have installed,
| outside large companies that must have cutting edge, or prosumers
| that have a incrementally better version than the masses?
|
| Are we a couple years away, a decade away, or something else?
| mdp2021 wrote:
| > _What does it take to go from here to a model on a pcie card
| or an m.2 card_
|
| It is already that.
|
| > _Will "intelligence" become much like a gpu_
|
| As an option among the implementations.
|
| > _Are we a couple years away_
|
| They could mass produce now, but it makes no sense at this rate
| of improvements in the models.
| sgc wrote:
| Thanks for answering. This is an 8b model, which are mainly
| curiosities outside niche tasks. I guess I am asking how far
| we are away from having today's more generally useful
| frontier model equivalents widely available for everyday
| users in their personal pcs/laptops via a single pcie or m.2
| drop in.
| mdp2021 wrote:
| > _how far we are away from having_
|
| It depends on AMD now. What was planned after the 8b was a
| ~30b, which is already sufficient (or more, when running at
| ultra-high speed).
| andix wrote:
| It would be quite ironic if this technology would render all
| those AI data centers practically useless. If the next step are
| just a much smaller amount of expensive chips, and the bottleneck
| becomes manufacturing those chips fast. Not building huge data
| centers and fighting for electrical power.
| linzhangrun wrote:
| Thinking that five or six years from now, Fable-level
| intelligence could be provided at 100x the current speed... makes
| me feel lost. I cannot imagine what the future will look like.
| DiscourseFan wrote:
| It will be cool but also violent and terrible.
| pizzaiolo wrote:
| So, like the present
| barbazoo wrote:
| With more wealth concentrates at the top, yes.
| bigyabai wrote:
| Presumably wealth would concentrate upwards even if AI
| was never made.
| sanex wrote:
| Yes it's a function of the monetary system. Absurd
| amounts of debt only certain people can access.
| DiscourseFan wrote:
| So you're saying we need to abolish to monetary system?
| __MatrixMan__ wrote:
| Like some kind of pimple, which we can pop from all
| sides. And then we'll build something different.
| Something that works.
| ilaksh wrote:
| Cerebras already runs large models like Kimi 2.6 or GLM at like
| 30x speed. 100 times is next year, not six years.
|
| You can actually test it out on their website, just imagine 3 x
| faster and maybe 15% smarter.
| keepupnow wrote:
| This.
| kllrnohj wrote:
| Cerebras is literally the entire wafer, so it can't get
| bigger. So where is the jump from 30x to 100x coming from?
| Node improvements only yield like 10-20% gains these days...
| ilaksh wrote:
| They have a next generation, I don't really know if it will
| be 3 x or what but I heard it was significantly better.
|
| Also there are other people innovating in hardware.
| AussieWog93 wrote:
| Could we not just make bigger wafers, if the technology
| called for it?
| kzrdude wrote:
| The investment in bigger machines at the fab might set
| you back billions. I don't know about the lithography
| technology either, how easy you can scale it to larger
| wafers?
| hhh wrote:
| we seem to be in a phase of spending trillions on the
| computer, so while it isn't likely, it isn't impossible
| rf15 wrote:
| There's so much you'd need to scale, it's even more
| expensive than the humble few billions you spend on a new
| fab. And, beyond R&D, the scale itself also makes it
| significantly more expensive. This is an energetic dead
| end, you would need a country to fully commit to making
| this one thing.
| Maxion wrote:
| Lithography is AFAIK the most expensive part. If you
| wanna scale to larger wafers I'd guess you're talking
| hundreds of billions and at least a decade. And even
| then, what will your yield be? Surface area grows to the
| square of the diameter of a circle, and so will errors.
| The larger your wafer goes the lower your yield will go,
| especially if you're going to use the whole wafer in one
| final product.
| techpression wrote:
| You also need to worry about yields, Apple, AMD etc can
| sell "bad" chips as lower core versions, if you're
| depending on whole wafer you have little room for error.
| kllrnohj wrote:
| Well they still just do the same thing GPUs do where bad
| clusters are disabled. There's going to be a balance
| there for how big the clusters are and how many tiers
| they'll offer, but that's a known problem which can be
| well modeled.
| cousinbryce wrote:
| Love the idea of discounts based on model error. "this
| one doesn't know what butterflies are, it's on sale for
| 8% off"
| kllrnohj wrote:
| Larger wafers has been attempted for over a decade now
| and doesn't seem to be going anywhere. See the 450mm
| wafer section of
| https://en.wikipedia.org/wiki/Wafer_(electronics)
| 1saadcodes wrote:
| Feels both unreal and dystopian. The speed at which these
| models are developing is very scary
| dyzone wrote:
| It tells me that they have some kind of insider knowledge that
| the models have hit their limits and won't be getting much
| better, and it makes sense economically speaking to just bake
| the current models and use them for the next 5-10 years. Looks
| like we're near the top of the S curve.
| mountainriver wrote:
| What would possibly tell you that?
| Maxion wrote:
| Expensive as fuck to make chips, only makes sense if you
| believe whatever model you're creating a chip out of will
| not become completely irrelevant in 5-10 years.
| niek_pas wrote:
| There's lots of use cases where the current models do
| fine though. A chip that can run a current cheap model at
| 100x would be amazing for things like detecting
| prohibited content on Facebook. You don't need the 2030
| equivalent of Fable for that, you need something that can
| cheaply process insane numbers of posts per day.
| IshKebab wrote:
| No, it makes sense if whatever model you're creating a
| chip out of will not be irrelevant in 5 years _even if it
| is much much faster & cheaper_.
|
| There are probably limited applications but not zero.
| mountainriver wrote:
| That's not what we are empirically seeing though. There
| is also no way of knowing that things can't be
| progressing for the next 5 years. No foundational model
| company would be spending the amount they are on research
| if they thought it wasn't going to pan out
| hahahaa wrote:
| A model you can run for practically no cost is a new
| proposition. It is the CPUification of AI. Sure there be
| supercomputers but you PC will be pretty super too.
| fsloth wrote:
| I'm not sure we are at the top. Swarmed lower tier models
| appear to scale up in competence. So something like a Fable
| swarm running at 100x the speed might provide surprises.
|
| But "top of the curve reached" feels like the likelier
| scenario.
| azernik wrote:
| No knowledge (insider or otherwise) is needed -- could just
| be a bet, hedged against all the other bets a company AMD's
| size makes.
| __MatrixMan__ wrote:
| Last month: agents spend 4 days on a hack, humans spend 3 weeks
| (so far) digging through the slop to figure out what happened
|
| Next time, one of those number will be smaller, and the other
| will likely be bigger. How long before the analysis side gets
| too overwhelming to bother with? Probably less than 6 years.
| threatripper wrote:
| It's not just speed, it will consume a lot less energy per
| token, maybe even more than 100x difference. And cost for a
| chip that runs that one model will also go down a lot once
| volume scales up. They will end up way cheaper than flexible
| GPU chips.
|
| I expect AI models chopped up into building blocks where 99.9%
| of the compute is fixed but glued together with flexible "fine
| tuning" layers that will adapt them to specific applications.
| Those kind of chips will run 99% of consumer AI and at some
| point be integrated into consumer devices.
| akie wrote:
| Ohhh that's a very interesting hypothesis. I think you might
| be right.
| matheusmoreira wrote:
| It makes me dizzy. I have no idea what is going to happen
| within even a year from now, can barely even imagine it.
|
| I'm trying to get the most out of it by redlining my AI
| subscriptions. Hopefully I'll manage to start a business in my
| niche. I don't even know if my niche will exist in the future.
| novaRom wrote:
| Not just you. Hassabis in a recent interview looked a bit
| nervous when talking about AGI is rather sooner than many
| expected.
| galaxyLogic wrote:
| I think the big news is that AMD is getting into memory-business
| so they won't be so dependent on Hynix and what have you. Memory
| is the bottleneck currently.
| nolist_policy wrote:
| AMD doesn't even have their own fans.
| OddMerlin wrote:
| Congrats to the Taalas gang.
| concraper wrote:
| A massive L for Canada
| galaxyLogic wrote:
| "... the chip serve Meta's Llama 3.1 8B at a blistering 16,960
| tokens a second -- when announced last February, that was 48x
| faster than Nvidia's GPUs and 8.5x faster than Cerebras'
| accelerators. "
| ford wrote:
| I've been showing people chatjimmy for months - it's incredible.
| Both reasoning and tool use generation scale with TPS. Imagine
| 100x more reasoning on a model, or 100x parallel tool uses.
| tech234a wrote:
| See also: Twitter statement from Taalas
| https://x.com/taalas_inc/status/2085458427757937097
| est wrote:
| Waiting for intelligence on a stick, plugin an USB, characters
| in, characters out.
|
| 100% local and no leaks.
| hliyan wrote:
| Question: we currently emulate neural networks by performing
| matrix math in synchronous clock CPU architectures. Would it not
| be better to abandon synchronization and etch neuron synapses
| directly in silicon, keeping only the weights variable? I think
| some researchers are pursuing this, but I forget what the
| approach is called.
| freakynit wrote:
| "Neuromorphic chips" .... and I have the exact same question in
| mind.
| Perz1val wrote:
| That'd be either analog computing or you're implementing
| floating point number math in each neuron. Maybe some kind of
| PWM calculations would be doable, do we have it researched or
| would be a new kind of math?
| Ey7NFZ3P0nzAe wrote:
| I don't understand what you'd etch instead of the weight?
| What's a neuron synapse to you that's not a weight?
| yunnpp wrote:
| I would've hoped the company stayed independent instead of being
| engulfed into a behemoth. I'd like to see more diversity in the
| hardware ecosystem, but I guess the economics of hardware
| manufacturing aren't there.
| kristianp wrote:
| I've been eagerly awaiting their 2nd gen HC2, which uses multiple
| chips to host a "mid sized reasoning" [1] model. Its due in
| summer according to the article, I wonder if it will ever be
| released in that form now.
|
| [1] https://www.forbes.com/sites/karlfreund/2026/02/19/taalas-
| la...
| NitpickLawyer wrote:
| > I wonder if it will ever be released in that form now.
|
| Yeah, I had the same thought. The key thing for them was the
| price point at which they could deliver a ~30B model. I would
| buy one today if it was ~1000$ and could run whatever the best
| 30B model is today, at those speeds advertised. Even if the
| model becomes superseded by model.5 in a few months, there's
| still a lot of things you can do with a "good enough" model for
| some tasks. And things like maj@x or generate 10 times and
| choose "at a glance" what you like (think frontend stuff) would
| be worth it.
|
| No idea if them selling to AMD is good or bad.
| num42 wrote:
| I have used chatjimmy before, it is incredibly fast, waiting for
| latest SOTA model on the chips in future. Great!
| jijji wrote:
| taalas is great for llama 3.x 8B models, really bad for one board
| serving Kimi K3, it seems like you would bottleneck at a few
| hundred tokens no matter what you do.... spreading the big model
| against multiple cards seems the only way to get into the 1k+
| tok/sec range. Another thing taalas is doing is masking the model
| weights into the silicon itself, not a flashable firmware, which
| would increase latency....
| 3836293648 wrote:
| They moved from HBM to dedicated silicon and only got a 48x speed
| up? That is so, so, so much less than I would've expected. Any
| numbers on how it scales?
| zkmon wrote:
| I guess the idea is, gains from inference speed could offset the
| cost of upgrading the chips to a new model when really required.
| I think general purpose models would consolidate and release
| frequency might flatten out, favoring this strategy.
| matheusmoreira wrote:
| > Once the chips are deployed you're stuck with that model.
|
| At least we can be sure that's the model we wanted. Service
| providers could be serving modified versions and nobody would
| ever know.
| analog31 wrote:
| Wow, we're heading back to mask-programmed ROMs. I'm feeling
| young again.
| ratsbane wrote:
| Smart move by AMD. Chatjimmy is very fast and not very good, but
| I think it might become very fast AND very good.
| yassa9 wrote:
| Can anyone imagine if a video generation model with the speed of
| ASICs baked into silicon ? real Sci-fi
| yigalirani wrote:
| what prevents amd to just do what they do without acquiring them?
| hugmynutus wrote:
| HN is rightly pointing out putting a model into an ASIC is kind
| of dumb.
|
| HN is failing to understand that AMD knows this well.
|
| Taalas has WO2025217724A1 pending and AMD wants that because it
| is immediately a function block they can sell to anyone doing FP
| math, since large (mostly) read only memory banks are ideally
| suited for that micro-code type stuff.
| christkv wrote:
| There is a big risk in etching a model into silicon like this. We
| are still evolving what small models look like and improving
| their performance. When do you decide to etch one into silicon
| knowing that right now an improved one can be 3 months away.
| roschdal wrote:
| Is this the singularity?
| tonyhart7 wrote:
| so in the future I can buy KIMI, GLM or whatever model that get
| "soldered" directly into GPU ????
|
| so instead of RTX xx70 series, I can buy xxTA that have kimi
| integrated ??? is that right ??
| spacedoutman wrote:
| I feel like this will be the end of Taalas, AMD has for the most
| part of its history always chosen the wrong options.
| jdub wrote:
| always? like amd64 vs itanium?
|
| (and AMD is still serious challenging Intel in x86 and GPUs
| today)
| rldjbpin wrote:
| given their specs [1] show ~100 mm^2 of silicon for each billion
| params (albeit in 6 nm process), what will come first: nuclear
| fusion or one of these that can handle the multi-trillon param
| models of today?
|
| [1] https://taalas.com/products/
| akie wrote:
| You're assuming no improvement on their current technology
| whazor wrote:
| According to my agent:
|
| > At 6 nm: 1T doesn't fit on one wafer.
|
| > At ~2 nm: 1T plausibly fits comfortably on one 300 mm wafer.
|
| But then again, 300B to 500B models are to this day also very
| valuable
| dave1010uk wrote:
| I'm surprised there's not more discussion about potential
| inflection points here. When technology gets faster, it opens up
| whole new classes of UX that were hard to predict
|
| For example, faster internet didn't mean being able to view 100x
| as many HTML4 web pages. It brought SaaS, streaming media and
| interactivity.
|
| I'm not good at predicting, but some ideas:
|
| 1. All information gets augmented in real time with personalised
| context.
|
| 2. AI interaction seems more like find-as-you-type than a back
| and forth.
|
| 3. AI produces many outputs to pick from. Either the human, AI,
| or another system can do the deciding.
|
| Even if it's last year's model, speeding up LLM inference could
| open up all sorts of opportunities.
| tybit wrote:
| Yes, this is a great point and it's even one that the Cerebras
| CEO spruiks for his own chips. Talaas has gone furthest on the
| spectrum here, so seems to have the most potential to evolve
| the use cases.
| amelius wrote:
| I think the most important direction will be: you spawn
| hundreds of agents at the same time, and let them work in a
| breadth-first search style. So you will not necessarily get
| your answers much faster, but they will be much more thoroughly
| researched. And if you do want faster responses, you can do
| that at the expense of quality.
| petra wrote:
| You could probably do real-time deep research using that
| method, and than summarize and ask questions about the
| results.
|
| That is a probably a significant jump in search quality for
| many queries, that people didn't take the time to research
| properly.
| overfeed wrote:
| 4. Pervasive , distributed dragnet surveillance under the
| misrepresentation that it's not a search until a human pulls
| the data. But a small on-device "E2E preserving" "safety" model
| that runs on your phone and snitches when illegal communication
| content is suspected.
|
| Edit: also consider centralized Room-641A-type surveillance
| when models summarize and/or flag all calls processed by public
| telephony
| dave1010uk wrote:
| I hope this doesn't come to BCI.
| bigyabai wrote:
| BCI will _only_ arrive once this style of monitoring can be
| enforced.
| bearjaws wrote:
| You could quite literally make an OS that generates your
| software ahead of you, like this VibeOS demo but lightning
| fast.
|
| https://youtu.be/7NfyZhV1dKM?t=53
|
| Imagine this demo but the apps render in real time generating
| real code.
|
| Obviously not valuable because we have OS today, but could be
| your companies "WorkOS"
| dmead wrote:
| robotic probes on mars could run science experiments and report
| results on their own.
|
| pacemakers could do deep analysis of heart signals and report
| problems.
| barbazoo wrote:
| Right, instead of just having more LLM conversations as they
| get faster and cheaper we'll find products having AI that never
| had it before.
|
| How about an actually smart thermostat that checks the weather
| and possibly makes decisions more like a human would i.e. tool
| calling, judgement, preference, history, personal plans.
|
| Sure we have thermostats and you can configure rules and data
| sources, hook up Google calendar, etc but it has to be all
| predetermined and breaks as soon as anything stops working. AI
| could make this less brittle. AI agents are more flexible.
| rbanffy wrote:
| My understanding is that the chip carries the neural network
| itself while the weights are loaded on top, so any model that
| uses that same network topology should be loadable on demand.
| bjackman wrote:
| No I think you are thinking of Etched/Sohu.
|
| Taalas' approach (at least for their demo'd product) is to bake
| the whole thing in completely. IIUC the optimiser can even see
| the weights while generating RTL. It's like there's an "uint8_t
| weights[] = " in the source code.
| fischeti wrote:
| They are loaded on top in the sense that they are contained in
| the upper layer(s) of the chip. So when you want to change the
| weights, you have to produce fewer masks for fabrication, which
| reduces cost and time to market.
| rbanffy wrote:
| Oh... That's not great - it'd be nice if it had a way to push
| updates without building a new chip.
|
| OTOH, maybe because of this our future cyberdecks will have
| cartridge ports.
| perarneng wrote:
| "Generated in 0.007s * 14,370 tok/s" - for some use cases this
| will be awesome
| perarneng wrote:
| On thing with this enormous speed is that you could create
| multiagent validation and correction and still have it super fast
| and achieve greater quality thanks to speed alone.
| Cort3z wrote:
| I believe this is big news. Speculations here on out: I imagine
| this being baked into consumer products, greatly increasing the
| local token capability for consumers. They will suck the cloud-
| oriented companies' milkshake. Most users do not need extremely
| capable models, they just need some automation to do better web-
| searches, and get simple facts etc. If it can do simple coding
| tasks too, but at thousands of tokens per second, in stead of
| tens or hundreds, the development will benefit so much. It will
| benefit AMD in other ways too. I imagine they can start selling
| physical chips, usb-drive like devices, that just does llm. If
| you want a newer, better, model, you simply go to a store and buy
| one. Need more capability, buy more drives. Similar to physx back
| in the day, but with usb-c and a smaller footprint.
| reqo wrote:
| One advantage that I see in models that are implemented as code
| is that they can quickly and cheaply be modified using LoRAs.
| What would the equivalent be in hardware? Another piece of
| hardware you would attach like adding a graphics card to a
| computer?
| trash_cat wrote:
| People are not talking enough how huge this is for robotics and
| IoT. Current robotics arhitectures are limited by tok/sec. How
| cares if its not a Fable model?
|
| This move undercuts NVIDIA directly.
| Zenst wrote:
| With AI models like Mixture of Experts, many of those experts
| will be the real target here, as polished, refined and little
| to no change, they become fine candidates for being locked into
| silicon. Who knows, add some SRAM in there and small changes to
| those experts could be carried out without needing new silicon.
|
| Maybe AI models may become reduced to a collection of tiles you
| add to a chips one day, maybe sooner for some areas as you say,
| motor control for balance, vision systems, speach recognition
| systems etc, broken down, for robotoics, much is already there
| and just cost of battery/power holding much back.
| Tade0 wrote:
| An "Expert" is really just an unfortunate name for what
| amounts to a dense part of a sparse matrix and that's also an
| oversimplification.
|
| It doesn't actually specialise in anything in particular that
| one can point to.
|
| For this reason you can really transfer them between models.
| 5555watch wrote:
| Is it still true? I'd assume you should be able to freeze
| the matrix and unfreeze an expert block, before feeding
| particularly chosen training data. Or that doesn't work?
| NamlchakKhandro wrote:
| no you're wrong. people are not talking about the load bearing
| seam that this strong decision has revealed towards veterinary
| care.
|
| the implications for mental health of pet rats is huge.
| luciana1u wrote:
| everyone's worried about the model being obsolete before the chip
| ships. nobody's worried about the chip being obsolete before the
| model ships.
| TechTechTech wrote:
| I think what will happen is what happened to something like 4K
| video decoding before where it ends up in silicon costing almost
| nothing to run extremely fast on device.
|
| "Good enough" LLM functionality (for the use case) will be on-die
| or on-chip for cars, appliances, etc. This will provide speeds of
| chatjimmy at a battery-level power consumption.
|
| Probably this will also happen for software engineering. Some
| usb-powered AI accelerator with Kimi K3 (and in future even
| better) performance running at 10K+ tokens/sec under 50W of power
| purchasable for almost no cost. Need a better model? Buy the new
| hardware. Old hardware is probably still fine for a lot of other
| use-cases. I expect China to be a big player here, it fits their
| open-model and hardware-manufacturing strategy.
| mixermachine wrote:
| Scaling a model on a chip is quite hard. ChatJimmy is based on
| Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That
| are 350x more parameters.
|
| I would expect that Gemma 4 E2B (approx 5.1 billion parameters)
| or maybe even Gemma 4 26 billion A4B at some point is running
| on a chip.
| formerly_proven wrote:
| It's quite telling that the 8B Taalas chip was already
| reticle-sized on TSMC N6. I mean, we're talking about a
| process that does ~100 MTr/mm2, ROM needs about one
| transistor per bit, but can probably be packed more densely
| than general logic. Something like, say, 150 megabit/mm2 is
| not a lot. N6 has a 850 mm2 reticle limit. This roughly
| tracks, the article says the chip has 8B parameters and
| apparently spends about half the area on ROM. There's a
| reason AI accelerators just use a ton of silicon area (each
| HBM3 die is >1000mm2 of silicon). I imagine this is not
| terribly viable unless they make it a lot more space
| efficient e.g. using MLC ROM if they don't already, or use
| stacked dies with a ROM-optimized process. And then we're
| back to not cheap, though reticle chips were never in the
| cheap area to begin with.
| Tuna-Fish wrote:
| Taalas exploits the low cardinality to store one 4-bit
| weight with one transistor. (They are using metal layer
| traces for the ROM, and connecting an access transistor to
| light up one of 16 options.)
|
| Their system is honestly very efficient for the weights,
| the problem is the KV-cache. That's why HC1 only supports
| such short context, they use SRAM for that and spend most
| of what's left of the die for it. The recent advancements
| that made attention more efficient are probably going to be
| very useful for them.
| jrflo wrote:
| Do you really need a KV cache if inference is that fast
| though?
| Tuna-Fish wrote:
| ... Yes. Quadratic is really bad for large enough n, and
| you need that big context for useful work.
| phonon wrote:
| A full wafer like Cerebras is about 60x that, and N2P has
| about 3x the transistor density. So right now it's
| technically feasible to etch a 1.4 trillion parameter
| model. So roughly DeepSeek-V4-Pro class. Imagine that
| running a factory, for example.
| briansm wrote:
| Cerebras have special techniques to work around etching
| errors / bad cores on their wafers. This is possible
| since their wafers are effectively hundreds of identical
| copies of redundant cores. Can't do that for a globally
| unique model.
|
| Etching failure in that situation would be like brain-
| damage in a human, all sorts of weird effects would start
| appearing.
| fulafel wrote:
| There's several ways to engineer around that as the
| errors are detectable. There's a big literature on how to
| trade off speed or transistors for error correction. [1]
|
| (Is Cerebras doing something novel? CPUs and memory
| blocks have been doing those things for a long time too,
| since the error rate is otherwise too high for normal
| size chips as well)
|
| [1] see eg https://www.vlsimentor.com/dft/redundancy-bisr
| to get some basic concepts
| wtallis wrote:
| > (each HBM3 die is >1000mm2 of silicon)
|
| Did you mean that each HBM3 _stack_ is that large? Because
| it only takes one glance to see that the memory chips are
| much smaller than reticle-sized GPUs they sit next to.
| Closi wrote:
| Scaling is definetly hard - but there is no absolute
| requirement to put huge flagship models into this technology
| (although it might be possible over time).
|
| A fairly dumb but FAST model has it's own totally distinct
| use-cases even if it can't be scaled in size. Think about a
| LLM-infused-Alexa where the response time is instant. Where
| you can request it looks at hotel options in Montreal, and it
| starts answering in half a second rather than a few minutes.
|
| Plus some sort of slow smart + fast dumb combo architectures
| might also work really well for different classes of
| problems.
| paulryanrogers wrote:
| > Where you can request it looks at hotel options in
| Montreal, and it starts answering in half a second
|
| Yet the answers will get outdated quickly whilst the
| silicon is fixed.
| fennecbutt wrote:
| >Yet the answers will get outdated quickly whilst the
| silicon is fixed.
|
| Bro is living in 2020 before rag was widely introduced.
| jermaustin1 wrote:
| I have found that for some personal prose-related projects,
| QWEN 3.6 35B A3B is an amazing model even quantized down to 4
| bits. I actually find it's "writing" style as a GM for an
| LLM-powered solo text adventure game, better than even some
| of the faster/dummer frontier models like GPT-5.6-Luna or
| Haiku 4.5, and it runs (slowly) on a 3090 with a 80k context.
|
| So I have faith in these embedded LLM chips when it comes to
| fun projects like that. I have not personally found my
| quantized QWEN good at agentic tasks, though, and it LOVES to
| make shit up when asking questions about documents in the
| prompt.
| dpedu wrote:
| 350x is only about 10-20 years of improvement, using CPU
| FLOPS as the benchmark.
| 8note wrote:
| wouldnt you want gpu, fpga, or dsp as the benchmark?
|
| its lots of parallel calculations, rather than one blazing
| fast one
| paulryanrogers wrote:
| Haven't CPUs largely plateaued? They're just getting
| bigger, more power hungry, and multiplying cores.
|
| Physics has hard limits and Moore's law is long dead.
| momojo wrote:
| There's certainly incentive to do so. And its _only_ an
| engineering problem haha.
| empath75 wrote:
| You could imagine different layers on different chips,
| though, i think...
| cmrdporcupine wrote:
| Consider a model like https://huggingface.co/nvidia/NVIDIA-
| Nemotron-Parse-2.0 which just came out. 0.9B parameters and
| very accurate for doing a very specific task: document
| features classification.
|
| Now imagine you have a chip which is just that model, but can
| do it at absolutely insane speed. Like tens of thousands of
| documents a second.
|
| Same for things like text-to-speech or speech-to-text. Think
| of the accessibility wins if subtitling becomes insanely
| accurate and fast and omnipresent.
|
| There are all sorts of domains like that, and the trend has
| been such that smaller models are getting smarter and
| smarter. If you can stick them in parking meters, traffic
| lights / street crossings, mobility aids, etc etc I just see
| so much potential win.
| Frannky wrote:
| Yes, it also opens up a faster recurring revenue model for
| hardware companies, faster model obsolescence than how often
| you change a computer, a server, or a GPU card. I hope they can
| figure out trillion-parameter models rapidly. Nvidia happened
| to be the best option for AI after building machines for
| graphics, so it makes sense they weren't the best idea from
| scratch for this specific use case. Especially given the scale
| of the demand and the possibility of recurring revenue, I hope
| a lot of smart people will try to solve it, compete with each
| other, and deliver us extremely fast and cheap intelligence.
|
| And one can say LLMs are not as smart as a human, but a lot of
| the reasoning humans do for product and service generation
| isn't smart at all--it's just a bit of fuzzy input/output plus
| some reasoning rules. And then if you hook a robot up to the
| LLM, you can get results in atoms instead of bits.
|
| I'm very excited about the future. I also hope it will stop
| money from flowing to bureaucrats who are incentived to keep
| the problems open to keep the money flowing, and instead
| facilitate sharing directly with the people (for example, no
| money to the state to solve homelessness--instead, spay instead
| with intelligence output to build a house and provide food as
| part of taxes.)
| joshspankit wrote:
| I think you're making another good point as well:
|
| Specific traces for specific inferencing will mean that some
| generations get deprecated. Look at H265.
| jameshart wrote:
| Commoditize your complements - still a winning strategy.
|
| If you make chips, you want models to be free.
| orangeberrytea wrote:
| Now the fun part, how will having an LLM in my washing machine
| help anything
| HPsquared wrote:
| It can ponder the meaning of its existence.
| joseda-hg wrote:
| "My Job Is To Open and Close Doors" [1]
|
| [1] - https://www.youtube.com/watch?v=49t-WWTx0RQ
| steve1977 wrote:
| It can insult the plumbing with "you're a dumb pipe".
| RobertDeNiro wrote:
| same way that having wifi does. by providing no actionable
| value, but boosting marketing materials
| imhoguy wrote:
| WiFi is fine for notification although Bluetooth would be
| enough to not get forgotten cloths stuck there for days.
|
| But why it calls home and why we have to create accounts to
| just get a notification.
| OneDeuxTriSeiGo wrote:
| Assistive technology.
|
| Imagine an energy efficient IC for a small multimodal model
| that can do voice to text, text to speech, question/answer,
| tool calling, and structured output.
|
| Wire that up to a microcontroller that parses the structured
| output to constrain the model (rather than giving the model
| direct hardware access).
|
| Now you have an assistive tech mode for supporting vision
| impaired users without requiring them to configure an app on
| their phone, pair devices, etc. And so now the user can just
| speak to the washing machine to tell it what to do.
|
| And because models are getting better and better at multi-
| language support, you can rely on a single model to cover a
| wide range of spoken languages. And therefore you don't need
| a bunch of variants of this chip for a single product line.
|
| TLDR this gives a path to replace "always online" and "wifi
| enabled" devices with fully on-device capabilities without
| being forced to abandon assistive technology support.
| mossTechnician wrote:
| I don't think that the ability to record and transcribe
| people's words will encourage manufacturers to remove Wi-Fi
| functionality. If anything, that would encourage them to
| keep it. Imagine the marketing possibilities.
| OneDeuxTriSeiGo wrote:
| sure but once the capability is there it's not unlikely
| we'll see regulatory pressure from regions like the EU or
| certain US states to ban or heavily limit sale of
| "collect all your data" wifi enabled devices.
|
| IMHO the only reason there's not been pushback from the
| EU already is because so much assistive tech is only
| possible with wireless connectivity currently. Once
| privacy conscious assistive tech is viable those
| crackdowns will probably be on the horizon even if only
| for the security risk associated with millions of
| internet connected devices with poor security posture.
| spicybright wrote:
| that would be so cool, but it'll be a long time to make the
| cost of an AI chip cheaper than a wifi enabled one if
| that's even possible.
| rnewme wrote:
| You load it, tell it what's in and it sets the program, tells
| you what it set and why. You approve and off it goes
| vel0city wrote:
| I do that today by turning a dial and pressing the start
| button.
| rnewme wrote:
| Not saying it's the superior way, but to be honest I
| wouldn't know how to start some washing machines without
| consulting manual first, having very dumb simple
| appliance myself.
| walrus01 wrote:
| I'm sure the manufacturer will figure out a way to:
|
| a) Have it display advertising on a screen on its front size
| (like some fridges now)
|
| b) Only work with approved laundry pods
|
| c) Other new revenue generating opportunities for its
| manufacturer
| pzo wrote:
| Washing machine was so inefficient whole time, it just heat
| water and spins. When you had to do by hand the hard decision
| is when to stop.
|
| I believe there is a lot we can improve in washing machines
| lanyard-textile wrote:
| Think bigger: A washing machine and dryer with an arm and a
| camera.
|
| It can sort your dirty and clean clothes before washing and
| after drying, based on your preferences.
|
| It defaults to common sense things like washing similar
| materials and weights, keeping delicates in their own load,
| grouping sensitive colors like red.
| fgblanch wrote:
| Tell the washing machine. "Washlexa I'm putting my gym
| clothes in. It was a hard workout I sweated a lot!"
|
| and the washer now knows what to do. No more temp, length,
| 2nd rinse proxy controls.
| pojzon wrote:
| And result is tiny little clothes, because 80C seemed right
| for the job.
|
| Washlexa:"Sorry, you were right I was supposed to use
| regular programme, but I used wrong one, do you want me to
| wash them again?"
| cptskippy wrote:
| https://www.youtube.com/watch?v=LRq_SAuQDec
| jhallenworld wrote:
| Never mind washing machine, how about a missile or a drone?
|
| Or a bomb.. "let there be light"
|
| https://www.youtube.com/watch?v=h73PsFKtIck
| bheadmaster wrote:
| It's possible in the future we will have Rick and Morty style
| AI in literally everything just because it's so easy to add it.
| Sentient Switchblade: "Hi Beth! You've gotten taller! Shall we
| resume stabbing?"
| rektomatic wrote:
| Nightblood? is that you?
| therealpygon wrote:
| I think this combined with a bit of memory and something like
| the "high-bandwidth flash" they just announced (if it works
| out), could be an interesting thing for some resident (burned)
| experts + active moe streamed from HBF.
|
| I expect one day having small lower power demand drive sized
| devices with proprietary burned-in models that are quite fast
| running on-device in robotics and such. Commoditizing LLMs via
| burned and locked hardware seems likely when LLMs have
| stabilized (when we reach a year between releases again) and
| the hardware is capable and "disposable" enough. "Buy a robot
| and upgrade it forever* (5 years) with newer models (sold
| separately)", at least until the planned hardware obsolescence
| that the "interface has changed to support newer hardware, so
| you'll need to upgrade (again) to use the latest features".
|
| The plans basically write themselves.
| joe_the_user wrote:
| I like "good enough" LLMs for search and quick trivia. But
| "cars, appliances, etc" is exactly where LLMs are between
| noxious and dangerous.
|
| My food processor could use a self-cleaning feature. It could
| only be made worse by some system that, IDK, changes the
| setting based on off-hand comments about "I don't know what his
| beef is.."
| woadwarrior01 wrote:
| I suspect this acquisition was a response to NVIDIA's acquisition
| of Groq.
|
| Cerebras is already public. AFAICT, there are 8 other startups in
| the space, some of which have mature products: Etched, d-Matrix,
| SambaNova, Tenstorrent, Positron, FuriosaAI, Rebellions and
| Fractile.
| nullc wrote:
| Might be an interesting motivation for looped LLMs to cut the
| gate count down. Perhaps even a collection of mixed programmable
| layers and baked layers in a loop.
| peter_d_sherman wrote:
| >"While the tech is blazing fast, if you hadn't already figured
| it out, it comes with a pretty substantial downside. Once the
| chips are deployed you're stuck with that model. Any change
| bigger than something like a LoRA adapter is going to require a
| re-spin of the chips, which is not only expensive but time-
| consuming."
|
| That is a downside to be sure, but from a pure business
| perspective, "that's not a bug -- it's a feature!"... from a pure
| business perspective it's the ability to sell and resell, to
| purchasing and re-purchasing customers, way into the future --
| that is, _recurring revenue_ from the perspective of the company
| being able to make those future recurring sales...
|
| In the above case, that company is AMD...
|
| (Also, on a related note, it would be interesting to see what
| open source / open hardware work has currently been done to
| offload LLM weights (and/or anything else that could be offloaded
| to silicon ASIC's) to FPGA's...)
| Iolaum wrote:
| TBH Taalas was a company I was existed about as a consumer. A
| dense model like gemma4-31b or qwen3.6-27b running at 10k t/s
| sounds like an awesome thing to have. Would be willing to pay GPU
| prices for it.
| harhargange wrote:
| My 7900XT runs it at 35 tokens per second.
| Tuna-Fish wrote:
| Note the k in his figure.
|
| Taalas HC1 was clocked at 17000 tokens/s.
| galeos wrote:
| Is there scope to implement ternary models using this approach to
| minimise die area of the model parameters?
| harhargange wrote:
| I have an AMD CPU as well as an AMD GPU. Their open-source
| philosophy and Linux compatibility, including a clean core
| distribution (No big.little for desktop) is what keeps me in. I
| think Taalas is in safe hands.
| ActionHank wrote:
| Big if it pans out.
|
| Economic and financial ripple effects would be huge aside from
| the obvious:
|
| - reduction in electricity usage
|
| - OpenAI / Anthropic are dead in the water unless they start to
| license their models to fabs.
|
| - Every single one of those GPUs that all of those massive data
| centers contain become paperweights.
| voxic11 wrote:
| You still need the GPUs for training new models though right?
| This is only suitable for inference.
| ActionHank wrote:
| For sure, but I strongly doubt that we would need nearly as
| many.
| Transformanshen wrote:
| I can actually see this being useful for fairly narrow workloads
| in dedicated devices where the model doesn't need to change very
| often and low-latency inference matters more than flexibility I
| don't see it replacing general-purpose GPUs but it seems like a
| reasonable option for that kind of workload
| 12912312 wrote:
| This is a sign of the AI bubble bursting. The 2000 bubble also
| saw a flood of hardware accelerators from SSL to packet switching
| to firewalls to graphics, most of which failed.
| brap wrote:
| While the demo is incredible, I think that in most practical use-
| cases, models aren't very useful without tools (search, code
| execution, etc.).
|
| Even if we assume reasoning latency drops to ~0ms (AFAIK this
| demo doesn't include reasoning at all), these use-cases will
| still remain relatively slow due to I/O of tool calls.
| faxmeyourcode wrote:
| Compaction, busting the cache, and other issues like that will
| lose significance when you're running at 15k tokens per second
| like chatjimmy. Very interesting to think about what will change
| in the future.
| bot41 wrote:
| I'm a big AI user, and have some minor experience training
| models, and follow some AI stories - but this is the coolest
| thing imo. The AI model/weights written directly on the chip!
| dabbz wrote:
| I see a lot of discourse about it being fast-to-deprecation. But
| I see it a different way personally.
|
| Modern LLMs are trying to do more with less. Focus on doing the
| right thing the first time. Even if we squeeze dumb LLMs, the
| significantly faster speed means quicker iterations. So a bad
| decision doesn't cost the time and inference costs that it cost
| before. It theoretically changes the scale of errant token spend.
|
| I compare it to the 1 thousand monkeys on a typewriter. In this
| case it's 1,000 monkeys with stale training data of everything
| ever written and the ability to search the web.
| vel0city wrote:
| I agree with this take in a lot of ways. If you slash the token
| cost and increase speed for each token 1,000x, who cares if it
| takes even 20x as many tokens to achieve the goal?
|
| And also, there are _lots_ of tasks where models today are fine
| with doing. If you think of these things like appliances, who
| cares if it 's not quite as powerful as the next generation? It
| was purchased to do a task, it still does that task very well.
| It feels like being in the 90s and asking "why buy a server
| today when they're going to be faster next year? Just keep
| renting mainframe time." Well maybe I just need a box to run
| our HR and payroll system, and this box manages to run it fine
| today.
| mafuy wrote:
| Sounds to me like you would hire 20 barely-paid interns instead
| of 2 competent programmers.
| ruined wrote:
| if it fits, it ships
| jedbrooke wrote:
| I'm surprised I haven't seen anyone mention video models yet. I
| don't know how many fps 17000 tok/s translates to exactly but
| it's gotta be a lot. Might make real time AI video possible.
|
| Now that I think about it, real time AI video might be a clear
| case of "You scientists were so preoccupied with whether you
| could or not, you forgot to ask if you should"
| graiz wrote:
| Take an open model and put it on chip, it becomes a dedicated
| sub-agent. You can still use a frontier model to check things or
| orchestrate but having 1000 sub-agents becomes very possible. I
| was already long AMD but just added to my position.
| adityazero wrote:
| There are market verticals where this makes a lot of sense.
| Embedded systems and IoT devices comes to mind.
|
| Even in data center space, I believe several layers can use fixed
| weights and remaining layers will compensate for the variations.
| Power savings will be huge so I think there is an incentive to do
| more of these.
| 5555watch wrote:
| Even for consumers.. Every enthusiast now wants those specced
| out rigs to play with LLMs. Make a nice chip for that, and it
| will reduce some pressure on consumer RAM demand
| hacker_homie wrote:
| So I guess,
|
| 1. This chip for an 8B model even if it was done at 5nm would
| still be twice the size of a conventional CPU die so what are the
| yields for this going to be like for even a 30B model?
|
| 2. They say 2 months but llama 3.1 was released 2024, ~2 years
| which is normal lead time for silicon, I suspect this would take
| longer if the architecture is not llama?
|
| 3. Can google do the same thing in house with their Gemma 4
| series (two year lead time puts Gemma 4 on silicon April 2028) ?
|
| I can see the benefit for hyper scalers but at the rate of model
| turn over does this type of investment make sense?
| harhargange wrote:
| Exactly, what will be the size of big models? Maybe they aren't
| targeting big models but where do they give size estimates?
___________________________________________________________________
(page generated 2026-08-08 09:01 UTC)