https://www.servethehome.com/next-gen-broadcom-pcie-switches-to-support-amd-infinity-fabric-xgmi-to-counter-nvidia-nvlink/ Refresh (1231 sec) Facebook Instagram Linkedin RSS TikTok Twitter Youtube * Forums * AI * Server + Server Systems + Server CPUs + Accelerators + Server Motherboards + Server Chassis + Other Components * 5G Edge * Storage * Networking * Workstation + Workstation Processors + Workstation Motherboards * Software + Operating Systems + Server Applications + Virtualization * Guides + Buyer's Guides + Tips + Top Hardware Components for TrueNAS / FreeNAS NAS Servers + Top Hardware Components for pfSense Appliances + Top Hardware Components for napp-it and Solarish NAS Servers + Top Picks for Windows Server 2016 Essentials Hardware + The DIY WordPress Hosting Server Hardware Guide Search [ ] Facebook Instagram Linkedin RSS TikTok Twitter Youtube Monday, December 11, 2023 * News * Storage Reliability + Raid Calculator + RAID Reliability Calculator | Simple MTTDL Model * Contact * About + Editorial and Copyright Policies * Subscribe * STH Forums Sign in Welcome! Log into your account [ ]your username [ ]your password [Login] Forgot your password? Get help Privacy Policy Password recovery Recover your password [ ]your email [Send My Password] A password will be e-mailed to you. [STH-Logo-96px] ServeTheHome Advertisement [INS::INS] [STH-Logo-48px] [STH-Logo-96px] * Forums * AI * Server + Server Systems + Server CPUs + Accelerators + Server Motherboards + Server Chassis + Other Components * 5G Edge * Storage * Networking * Workstation + Workstation Processors + Workstation Motherboards * Software + Operating Systems + Server Applications + Virtualization * Guides + Buyer's Guides + Tips + Top Hardware Components for TrueNAS / FreeNAS NAS Servers + Top Hardware Components for pfSense Appliances + Top Hardware Components for napp-it and Solarish NAS Servers + Top Picks for Windows Server 2016 Essentials Hardware + The DIY WordPress Hosting Server Hardware Guide [ ][Search] Home AI Next-Gen Broadcom PCIe Switches to Support AMD Infinity Fabric XGMI to Counter... * AI * Networking Next-Gen Broadcom PCIe Switches to Support AMD Infinity Fabric XGMI to Counter NVIDIA NVLink By Patrick Kennedy - December 11, 2023 1 Facebook Twitter Pinterest Linkedin Email Print Copy URL AMD EPYC Infinity Fabric Topology MappingAMD EPYC Infinity Fabric Topology Mapping Last week we covered the AMD Instinct MI300X GPU and MI300A APUs Launched for AI Era. During the event, we learned that AMD is going much broader with its XGMI technology, called Infinity Fabric. Broadcom announced that its next-generation PCIe switches will support XGMI/ Infinity Fabric on stage. There were also rumblings at the event around XGMI-connected NICs which would be the next step for RDMA NICs. Next-Gen Broadcom PCIe Switches to Support AMD Infinity Fabric XGMI to Counter NVIDIA NVLink During the event, AMD showed its massive GPUs and APUs dubbed the AMD Instinct MI300X and MI300A respectively. AMD MI300 Event Wrap Up SlideAMD MI300 Event Wrap Up Slide It is one effort to build a server with eight OAM accelerators stick them in a UBB and make an 8-way accelerator server platform. It is another to scale that server platform to a cluster of servers to do AI training. AMD AI Day 2023 Scaling Beyond The 8 GPU ServerAMD AI Day 2023 Scaling Beyond The 8 GPU Server At the event, AMD said that it is putting its weight behind Ethernet, and specifically the Ultra Ethernet Consortium to use Ethernet instead of Infiniband as the interconnect of choice for AI clusters. UltraEthernet For Scale Out AI TrainingUltra Ethernet For Scale Out AI Training AMD then said that it is opening up its Infinity Fabric ecosystem to new partners. AMD Expanding Access To The Infinity Fabric And XGMI EcosystemAMD Expanding Access To The Infinity Fabric And XGMI Ecosystem Infinity Fabric is perhaps best known as the interconnect that AMD uses between processors in EPYC servers. AMD EPYC 9004 Genoa Infinity Fabric OverviewAMD EPYC 9004 Genoa Infinity Fabric Overview One of the really neat capabilities of AMD Infinity Fabric/ XGMI controllers is that they can serve multiple functions. AMD's I/O controllers can do things like handle package-to-package connectivity as Infinity Fabric, PCIe Gen5 for cards, and CXL. On stage at the event, Jas Tremblay, Vice President and General Manager of the Data Center Solutions Group at Broadcom said that its next-generation PCIe switches will support XGMI/ Infinity Fabric. Broadcom Announcing It Will Support XGMI Infinity Fabric In Next Gen PCIe Switches At AMD AI Day 2023Broadcom Announcing It Will Support XGMI Infinity Fabric In Next-Gen PCIe Switches At AMD AI Day 2023 That is an important tidbit. We expect the next-generation of PCIe switches to also start looking at features like CXL and have shown the switches from companies like XConn for a number of years. XConn SC50256 CXL 2.0 Switch Chip FMS 2022 1XConn SC50256 CXL 2.0 Switch Chip FMS 2022 1 It seems like Broadcom is not just looking at PCIe and CXL, but it also has plans to have switches for things like Infinity Fabric. At a system level, that is a big deal. NVIDIA HGX V100, A100, and H100 systems have one internal level of NVSwitches to provide an interconnect within a chassis, and then an external link for external switches. NVIDIA also briefly had a 16-GPU configuration in the NVIDIA DGX-2 with tray-to-tray NVSwitch links. Inspur NF5488A5 NVIDIA HGX A100 8 GPU Assembly Larger NVSwitch CoolersInspur NF5488A5 NVIDIA HGX A100 8 GPU Assembly Larger NVSwitch Coolers For AMD, this is a big deal. A switched topology will allow more than eight GPUs per node without having to use less efficient PCIe signaling. We also heard another nugget at the show. Imagine not just the CPU and GPU communication happening over XGMI/ Infinity Fabric. Instead, imagine a world with XGMI-connected NICs where the NIC was on the same coherent fabric. If AMD intends to extend its AI training clusters via Ultra Ethernet, then instead of having to go CPU/ GPU to PCIe with RDMA transfers over the PCIe Ethernet NIC, imagine having the XGMI NIC sitting on the Broadcom XGMI switch with the CPU and multiple GPUs. That is a fascinating concept. Final Words While this is some future technology, AMD needs it. Its story of scaling beyond 8-GPU machines is Ethernet. Ultra Ethernet will help, but our best guess is that with the consortium forming in 2023, we are still some time off until Silicon on that side. AMD's 2024 sales forecast of $2B (credit Max at Reuters) feels very low. AMD is shipping MI300X systems from some OEMs this month, and we expect at least one OEM to have $300M+ of MI300 bookings before Christmas 2023 and those systems are not for Microsoft. If AMD can deliver the MI300X chips, it is going to blow out a $2B forecast in 2024. With so much hardware being deployed, we hope Broadcom can deliver its next-gen Infinity Fabric PCIe switch sooner rather than later. [INS::INS] * TAGS * amd * Broadcom * Infinity Fabric * NVIDIA * NVLink * NVSwitch * Ultra Ethernet * XGMI Facebook Twitter Pinterest Linkedin Email Print Copy URL Previous articleMarvell Octeon CN103 DPU 50Gbps Encryption on One Core Patrick Kennedy https://www.servethehome.com Patrick has been running STH since 2009 and covers a wide variety of SME, SMB, and SOHO IT topics. Patrick is a consultant in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about server, storage and networking, building blocks. If you have any helpful information please feel free to post on the forums. Facebook Linkedin Twitter RELATED ARTICLESMORE FROM AUTHOR Marvell Octeon VN103 DPU Chip Marvell Octeon CN103 DPU 50Gbps Encryption on One Core Google Cloud TPU V5p Google TPU v5p AI Chip Launches Alongside Gemini AMD Instinct MI300 Launch Cover AMD Instinct MI300X GPU and MI300A APUs Launched for AI Era 1 COMMENT 1. emerth December 11, 2023 At 2:43 pm Excellent. This cannot happen fast enough. LEAVE A REPLY [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] Please enter your comment! [ ] Please enter your name here [ ] You have entered an incorrect email address! Please enter your email address here [ ] [ ]Save my name, email, and website in this browser for the next time I comment. [ ] [ ]Sign me up for the STH newsletter! [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] This site uses Akismet to reduce spam. Learn how your comment data is processed. --- Advertisements --- [INS::INS] [INS::INS] [STH-Logo-96px] ABOUT US ServeTheHome is the IT professional's guide to servers, storage, networking, and high-end workstation hardware, plus great open source projects. Advertise on STH DISCLAIMERS: We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites. Please be advised of our data usage for machine learning and AI inference and training purposes which can be found here along with our other editorial and copyright policies. FOLLOW US Facebook Instagram Linkedin RSS TikTok Twitter Youtube (c) 2009-2023 Loyolan Ventures, LLC Refresh (1231 sec)