[HN Gopher] Meta MTIA v2 - Meta Training and Inference Accelerator
       ___________________________________________________________________
        
       Meta MTIA v2 - Meta Training and Inference Accelerator
        
       Author : _yo2u
       Score  : 150 points
       Date   : 2024-04-10 15:16 UTC (7 hours ago)
        
 (HTM) web link (ai.meta.com)
 (TXT) w3m dump (ai.meta.com)
        
       | modeless wrote:
       | Intel Gaudi 3 has more interconnect bandwidth than this has
       | memory bandwidth. By a lot. I guess they can't be fairly compared
       | without knowing the TCO for each. I know in the past Google's TPU
       | per-chip specs lagged Nvidia but the much lower TCO made them a
       | slam dunk for Google's inference workloads. But this seems pretty
       | far behind the state of the art. No FP8 either.
        
         | leetharris wrote:
         | They are different architectures optimized for different
         | things.
         | 
         | From the Meta post: "This chip's architecture is fundamentally
         | focused on providing the right balance of compute, memory
         | bandwidth, and memory capacity for serving ranking and
         | recommendation models."
         | 
         | Optimizing for ranking/recommendation models is very different
         | from general purpose training/inference.
        
           | modeless wrote:
           | Yeah, it may fit their current workload perfectly, but it
           | doesn't seem very future proof with the limited bandwidth.
           | Given how fast ML is evolving these days I question if it
           | makes sense to design and deploy a chip like this. I guess
           | they do have a very large workload that will benefit
           | immediately.
        
             | tony_cannistra wrote:
             | Don't mean to single you out at all, but I find this
             | comment to be a great example of how the "ML Hype" is
             | perceived by a certain segment folks in our industry.
             | 
             | The development of this chip shows that it doesn't (and
             | shouldn't!) matter to the ML teams at Meta how 'fast ML is
             | evolving.'
             | 
             | Indeed what it demonstrates is that a huge, global,
             | trillion-dollar business has operationalized an existing ML
             | technology to the extent that they can invest into, and
             | deploy, customized hardware for solving a business problem.
             | 
             | How ML "evolves" is irrelevant. They have a system which
             | solves their problem, and they're investing in it.
        
               | airstrike wrote:
               | Not to mention the capabilities they developed by
               | actually creating this and what they'll be able to do
               | next thanks to this experience.
               | 
               | You've gotta learn to walk before you can run
        
               | prpl wrote:
               | To me, it's bizarre to see the HPC mindset taking hold
               | again after the cloud/commodity mindset dominated the
               | last 16 years.
               | 
               | You don't always need a Ferrari to go to the store
        
               | thorncorona wrote:
               | WDYM by HPC mindset?
        
               | rfoo wrote:
               | "The only meaningful benchmark in the world is LAPACK and
               | only larger than ever monolithic problem instances
               | matter, I don't know what you're talking about,
               | 'embarrassingly parallel'? What a silly word! Serving web
               | requests _concurrently_? Good for you, congratulations,
               | but can you do parallel programming? "
               | 
               | Sorry if this make anyone feels bad. It certainly made
               | myself uncomfortable typing it out though.
        
               | prpl wrote:
               | Roughly this. Part of it is performance fetish. Part of
               | it is one architecture for every purpose. I can't tell
               | you how many times I've seen people run embarrassingly
               | parallel jobs coordinated by MPI on a Cray - because
               | somebody spent all that money on that machine. Don't
               | forget about Bell prize outages.
        
               | janalsncm wrote:
               | In their defense, it's because the article is
               | (understandably) sparse on details about what makes the
               | requirements of their ranking models different from image
               | classification or LLMs. Unless you work in industry it's
               | unlikely you will have heard of DeepFM or ESMM or
               | whatever Meta is using.
               | 
               | And building out specialized hardware _does_ lock you in
               | to a certain extent. Want to use more than 128GB of
               | memory? Too bad, your $10B chip doesn't support that.
        
               | sangnoir wrote:
               | > Want to use more than 128GB of memory? Too bad, your
               | $10B chip doesn't support that.
               | 
               | Which is probably why Meta is also buying the biggest
               | Nvidia datacenter cards by the shipload. There is no need
               | to run inference for a small model - say for a text-ad
               | recommendation system - on an H100 with attendant
               | electricity and cooling costs.
        
             | Aurornis wrote:
             | > Yeah, it may fit their current workload perfectly, but it
             | doesn't seem very future proof
             | 
             | It's custom silicon designed for a specific, known
             | workload. It's not designed to be a general purpose part or
             | to be future proofed for unknown future applications.
             | 
             | When a new application comes along with new requirements,
             | the teams will use their experience to create a new chip
             | targeting that new application.
             | 
             | That's the great part about custom silicon: You're not
             | hitting general specs for general applications that you may
             | not even know about yet. You're building one very specific
             | thing to do a very specific job and do it very well.
        
               | noiseinvacuum wrote:
               | Right and they have a LOT of GPUs from Nvidia for handle
               | the unknown. Custom silicon for custom workloads seems
               | like a good strategy specially considering the
               | capabilities that the team will develop along the way.
        
               | giantrobot wrote:
               | Offloading a known workload to a custom chip can also
               | save a lot on operations costs, particularly power.
               | Facebook is interested in _workload_ operations per watt
               | rather than raw floating point operations per watt. A GPU
               | might have better raw specs but if the whole GPU package
               | has worse workload ops per watt, a custom chip is likely
               | better.
               | 
               | At Facebook's scale the spherical cow raw performance
               | stats don't matter nearly as much as real world workloads
               | per ops dollar. They can also repurpose their GPUs to
               | other workloads and let their custom chips handle the
               | boring baseline stuff.
        
           | janalsncm wrote:
           | Translation: you don't need to serve 96 layer transformers
           | for ranking and recommendation. You're probably using a
           | neural net with around 10-20 million parameters. But it needs
           | to be fast and highly parallelizable, and perhaps perform
           | well in lower precisions like f16. And it would be great to
           | have a very large vector LUT on the same chip.
        
             | samspenc wrote:
             | Is there a better way to compare performance across these
             | high-end chips? The only comparable numbers I was able to
             | find were the TFLOPS.
             | 
             | Meta seems to be reported these numbers for this v2 chip:
             | 708 TFLOPS/s (INT8) (sparsity)         354 TFLOPS/s (INT8)
             | 
             | And I see Nvidia reporting these numbers for its latest
             | Blackwell chips
             | https://www.anandtech.com/show/21310/nvidia-blackwell-
             | archit...                   4500 T(FL)OPS INT8/FP8 Tensor
             | 
             | Am I understanding correctly that Nvidia's upcoming
             | Blackwell chips are 5-10x faster than this one Meta just
             | announced?
        
         | chabons wrote:
         | > Intel Gaudi 3 has more interconnect bandwidth than this has
         | memory bandwidth.
         | 
         | LPDDR5 vs HBMe2. I'm guessing there's a 2-5x price difference
         | between those, but even so it's an _interesting_ choice, I don
         | 't know any other accelerators which spec DDR. But yeah,
         | without exact TCO numbers it's hard to compare exactly.
        
         | cma wrote:
         | Only 48MB of SRAM on Gaudi 3 per die (96 MB across both) vs
         | 256MB here maybe increases the memory bandwidth needs for
         | Gaudi. Way different power consumption too.
        
         | chessgecko wrote:
         | Also its at 90 watts vs 900 watts for gaudi 3, the flops/mem bw
         | per watt is much more comparable.
        
           | moffkalast wrote:
           | It would be interesting if this could be made into a
           | reasonably priced (lmao) card for home inference if they
           | intend to mass produce it.
           | 
           | Can't imagine any other reason other than cost as to why they
           | went with LPDDR5, LPDDR5X has more bandwidth and GDDR6 has
           | even more.
        
             | chessgecko wrote:
             | they didn't use GDDR cause they wanted the memory capacity
             | which is really important for recommendation models. But I
             | totally agree that this is a sort of perfect cost/perf per
             | watt point for a home setup. I really hope they do it, if
             | not for this one at least for v3.
        
           | modeless wrote:
           | With high end chips like that it's often possible to get
           | dramatically better efficiency by running it at less than
           | peak power consumption, like 90% performance at 50% power or
           | something like that. It's hard to compare the numbers in a
           | fair way.
        
       | sroussey wrote:
       | Pretty large increase in performance over v1, particularly in
       | sparse workloads.
       | 
       | Low power 25W
       | 
       | Could use higher bandwidth memory if their workloads were more
       | than recommendation engines.
        
         | tasty_freeze wrote:
         | First gen was 25W. The new one is 90W.
        
           | sroussey wrote:
           | Ah, thanks for the correction.
           | 
           | Still relatively low compared to GPUs.
        
       | xnx wrote:
       | My mind still boggles that a BBS+ads company would think it needs
       | to design its own chips.
        
         | falcor84 wrote:
         | "Depending on how you want to think about it, it was funny or
         | inevitable or symbolic that the robotic takeover did not start
         | at MIT, NASA, Microsoft or Ford. It started at a Burger-G
         | restaurant ..."
         | 
         | https://marshallbrain.com/manna1
        
           | pksebben wrote:
           | dangit, I've got things I should be doing. Posting
           | interesting stories during business hours _continues
           | grumbling incoherently_
        
           | searchableguy wrote:
           | https://www.ycombinator.com/companies/ofone/jobs/u2E2fCX-
           | fou...
           | 
           | I saw this YC startup ad right after I finished reading this.
        
         | rsynnott wrote:
         | Well, the first commercial computer was created by a company
         | whose primary business was running cafes...
         | https://en.wikipedia.org/wiki/LEO_(computer)
        
         | libria wrote:
         | Or that an online bookseller would try to rent out compute.
        
         | okdood64 wrote:
         | They literally print money; smart move for them to make this
         | investment imo.
        
         | hackerlight wrote:
         | You're thinking like a startup founder where you should only
         | focus on innovating your main product. FB is a mature company
         | where some vertical integration can make sense.
        
       | jsheard wrote:
       | I like the interactive 3D widget showing off the chip. Yep, that
       | sure is a metal rectangle.
        
         | whilenot-dev wrote:
         | Really annoys me that the loading animation of these
         | before-/after-images doesn't finish on firefox and that it
         | won't let me drag the knob with the separator. ...no "Under the
         | hood" for me.
        
           | a_wild_dandan wrote:
           | Dragging the top-left corner works, for some reason. Really
           | bizarre UI issue.
        
         | huevosabio wrote:
         | jajajaj I thought the same! I thought maybe someone with
         | hardware experience can make a sense of this?
        
         | TulliusCicero wrote:
         | Exactly what I was thinking. Like showing off a model of a
         | blank DVD.
        
       | teaearlgraycold wrote:
       | Still seems pretty primitive. Very cool though.
       | 
       | I can only imagine the lack of fear Jensen experiences when
       | reading this.
        
         | airstrike wrote:
         | It would be foolish to underestimate the long term capabilities
         | of a sufficiently funded and driven competitor
        
         | moffkalast wrote:
         | _adjusts black leather jacket_ "Look at what they need to mimic
         | a fraction of our power."
        
       | jrgd wrote:
       | I find it weird that not everyone agree Meta and Facebook and
       | social networks in general are doing some good the the society
       | and our democracies; yet they manage to spend incredible amount
       | of money/energy/time to develop solutions to problems we aren't
       | exactly sure are worth solving...
        
         | ixaxaar wrote:
         | If all this turns out to be useless, burning their cash for
         | nothing seems like a great way to accelerate tech while going
         | down. I guess that would actually be a positive thing.
        
         | pptr wrote:
         | What is worth solving in your opinion? Should they not make
         | their service more efficient?
         | 
         | I assume this helps reduce their server and electricity costs.
         | At a certain scale these things pay off.
        
       | chessgecko wrote:
       | I thought MTIA v2 would use the mx formats
       | https://arxiv.org/pdf/2302.08007.pdf, guess they were too far
       | along in the process to get it in this time.
       | 
       | Still this looks like it would make for an amazing prosumer home
       | ai setup. Could probably fit 12 accelerators on a wall outlet
       | with change for a cpu, would have enough memory to serve a 2T
       | model at 4bit and reasonable dense performance for small training
       | runs and image stuff. Potentially not costing too much to make
       | either without having to pay for cowos or hbm.
       | 
       | I'd definitely buy one if they ever decided to sell it and could
       | keep the price under like $800/accelerator.
        
         | buildbot wrote:
         | I suppose it might, there are not a lot of details (what kind
         | of sparsity for example?) about what they mean in terms of INT8
         | support - it could be MXINT8, or something else.
         | 
         | Glad someone was thinking the same thing I was though!
        
           | chessgecko wrote:
           | its gotta be that 2/4 sparsity that everyone has, but I
           | haven't seen used anywhere right? If they put it in though
           | they must be using it, but I'm not sure for what. And without
           | details I think its a good bet that int8 is the standard
           | int8.
           | 
           | Wishful thinking maybe they'll announce selling it with the
           | giant llama3 cause there's no good, cheap way to inference
           | something like that at home at the moment and this could
           | change that.
        
       | duchenne wrote:
       | Is it possible to buy it?
        
       | throwaway48476 wrote:
       | It's interesting that they are not separating training and
       | inference.
        
         | noiseinvacuum wrote:
         | This is specifically designed for inference for recommendations
         | models. It's not for LLM training or inference.
        
       | mlsu wrote:
       | Certainly an interesting looking chip. It looks like it's for
       | recommendation workloads. Are those workloads very specific, or
       | is there a possibility to run more general inference (image,
       | language, etc) on this accelerator?
       | 
       | And, they mention a compiler in PyTorch, is that open sourced? I
       | really liked the Google Coral chips -- they are perfect little
       | chips for running image recognition and bounding box tasks. But
       | since the compiler is closed source it's impossible to extend
       | them for anything else beyond what Google had in mind for them
       | when they came out in 2018, and they are completely tied to
       | Tensorflow, with a very risky software support story going
       | forward (it's a google product after all).
       | 
       | Is it the same story for this chip?
        
       | ein0p wrote:
       | Come on, Zuck, undermine Google Cloud and take NVIDIA down a few
       | pegs by offering this for purchase in good quantities.
        
       | bevekspldnw wrote:
       | Pretty fascinating they mention applications for ad serving but
       | not Metaverse.
       | 
       | I feel like Zuck figured out he's just running an ads network,
       | the world is a long way anway from some VR fever dream, and to
       | focus on milking each DAU for as many clicks as possible.
        
       ___________________________________________________________________
       (page generated 2024-04-10 23:01 UTC)