https://www.servethehome.com/ethernet-ssds-hands-on-with-the-kioxia-em6-nvmeof-ssd/ Refresh (1231 sec) * Forums * AI * Server + Server Systems + Server CPUs + Accelerators + Server Motherboards + Server Chassis + Other Components * 5G Edge * Storage * Networking * Workstation + Workstation Processors + Workstation Motherboards * Software + Operating Systems + Server Applications + Virtualization * Guides + Buyer's Guides + Tips + Top Hardware Components for FreeNAS NAS Servers + Top Hardware Components for pfSense Appliances + Top Hardware Components for napp-it and Solarish NAS Servers + Top Picks for Windows Server 2016 Essentials Hardware + The DIY WordPress Hosting Server Hardware Guide Search [ ] Wednesday, April 20, 2022 * News * Storage Reliability + Raid Calculator + RAID Reliability Calculator | Simple MTTDL Model * Contact * About + Editorial and Copyright Policies * Subscribe * STH Forums Sign in Welcome! Log into your account [ ]your username [ ]your password [Login] Forgot your password? Get help Privacy Policy Password recovery Recover your password [ ]your email [Send My Password] A password will be e-mailed to you. STH ServeTheHome Advertisement [INS::INS] STH STH * Forums * AI * Server + Server Systems + Server CPUs + Accelerators + Server Motherboards + Server Chassis + Other Components * 5G Edge * Storage * Networking * Workstation + Workstation Processors + Workstation Motherboards * Software + Operating Systems + Server Applications + Virtualization * Guides + Buyer's Guides + Tips + Top Hardware Components for FreeNAS NAS Servers + Top Hardware Components for pfSense Appliances + Top Hardware Components for napp-it and Solarish NAS Servers + Top Picks for Windows Server 2016 Essentials Hardware + The DIY WordPress Hosting Server Hardware Guide [ ][Search] Home Storage Ethernet SSDs - Hands-on with the Kioxia EM6 NVMeoF SSD * Storage Ethernet SSDs - Hands-on with the Kioxia EM6 NVMeoF SSD By Patrick Kennedy - April 20, 2022 3 Facebook Twitter Pinterest Linkedin Email Print Koixia EM6 Cover For WebKoixia EM6 Cover For Web Today we have a fairly exciting piece. This is our first hands-on with a new class of SSDs designed for a different scale-out deployment model. The Kioxia EM6 is a NVMeoF SSD that uses Ethernet and RDMA instead of PCIe/ NVMe, SAS, or SATA to connect to the chassis. We get hands-on to show you why this is very cool technology. Ethernet SSD Video Version We have a video version of this one as well. In the video, we are able to show what the screens look like and show blinking videos and features like hot-swapping the drives. As always, we suggest opening this in a new browser/ tab for the best viewing experience. Kioxia EM6 NVMeoF SSD in the Ingrasys ES2000 First off, the biggest change in this entire solution is the Kioxia EM6 SSD. This may look like a normal 2.5'' SSD, but there is a big difference, its I/O is Ethernet rather than traditional PCIe/ NVMe. One can see that the Ingrasys ES2000 sled has two EDSFF connectors on it. This is to maintain a common design with traditional NVMe products, but it highlights another feature. These are dual-port NVMeoF SSDs for redundancy. That is an important feature to allow them to compete in markets traditionally serviced by dual-ported SAS SSDs. Ingrasys ES2000 And Kioxia EM6 In TrayIngrasys ES2000 And Kioxia EM6 In Tray We looked at the Ingrasys ES2000 for Kioxia EM6 NVMe-oF SSDs at SC21. This is the first time that we got to actually use the system though. Ingrasys ES2000 And Kioxia EM6 24 Bay FrontIngrasys ES2000 And Kioxia EM6 24 Bay Front In the Kioxia lab, we got to use an older revision chassis but one can see it is a 24-bay 2U that looks like many other servers and storage shelves we have seen over the years. Ingrasys ES2000 Kioxia EM6 NVMeoF SSD Chassis FrontIngrasys ES2000 Kioxia EM6 NVMeoF SSD Chassis Front We show this in the video, but we can actually hot-swap the EM6 SSDs. The EDSFF connectors are designed for hot-swap, and Ethernet networks, of course, are designed to handle devices going online/ offline. Ingrasys ES2000 Kioxia EM6 NVMeoF SSD Chassis 3Ingrasys ES2000 Kioxia EM6 NVMeoF SSD Chassis 3 Here is a look at the backplane that the drives plug into. As one can see, we have EDSFF connectors but only one set even though the drive trays have two connectors. Ingrasys ES2000 And Kioxia EM6 Controller EDSFF BackplaneIngrasys ES2000 And Kioxia EM6 Controller EDSFF Backplane This is because there are two in each chassis. Here is a look at the switch node under a plexiglass cover. Ingrasys ES2000 And Kioxia EM6 Controller Node InstalledIngrasys ES2000 And Kioxia EM6 Controller Node Installed Looking at the rear of the system, we can see the two switches. This particular system only has the top node connected. Also, as a fun note, since this is an Ethernet switch node, we can have 1-6 connections and in this system the one connection being used is 100GbE. Ingrasys ES2000 Kioxia EM6 NVMeoF SSD Chassis Rear 2Ingrasys ES2000 Kioxia EM6 NVMeoF SSD Chassis Rear 2 One can also see that each network switch tray has a USB port, an out-of-band management port, and a serial console port. While they may look like servers, these are switches. Ingrasys ES2000 Kioxia EM6 NVMeoF SSD Chassis Rear 1Ingrasys ES2000 Kioxia EM6 NVMeoF SSD Chassis Rear 1 Inside the system, we can see the Marvell 98EX5630 switch chip covered by the big heatsink. Marvell also makes the NVMe to Ethernet controllers in the EM6 SSDs. Ingrasys ES2000 Switch Tray 1Ingrasys ES2000 Switch Tray 1 There is also a control plane for the switch under the black heatsink. This is powered with an Intel Atom C3538 CPU along with a M.2 SSD and 8GB of DRAM. Ingrasys ES2000 Switch Tray 2Ingrasys ES2000 Switch Tray 2 These switches, stacked one above the other allow each drive to access both switch networks simultaneously providing a redundant path to the drives. For those wondering, there is a Marvell 88SN3400 adapter card that connects a traditional NVMe SSD to Ethernet so it can be used in this chassis. Ingrasys ES2000 And Kioxia EM6 NVMe To 25GbE EDSFF BridgeIngrasys ES2000 And Kioxia EM6 NVMe To 25GbE EDSFF Bridge Next, let us get hands-on and see some of the fun things we can do with the Kioxia EM6 SSDs. [INS::INS] 1 2 * TAGS * Ingrasys * Kioxia * NVMe * NVMe-oF * NVMeoF * SSD Previous articleFS QSFP28-100G-SR4 v. QSFP28-100G-IR4 Differences Patrick Kennedy https://www.servethehome.com Patrick has been running STH since 2009 and covers a wide variety of SME, SMB, and SOHO IT topics. Patrick is a consultant in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about server, storage and networking, building blocks. If you have any helpful information please feel free to post on the forums. RELATED ARTICLESMORE FROM AUTHOR Amphenol SlimSAS Internal And MiniSAS HD External Cable Assemblies Storage What is 24G SAS and How does SAS-4 Fit in the Data Center Toshiba Kumoscale STH Cover Storage Kioxia Kumoscale 3.20 Released HP EX900 1TB Storage HP EX900 1TB NVMe SSD Review 3 COMMENTS 1. Flav April 20, 2022 At 12:05 pm Any indication on pricing and availability? 2. Carl April 20, 2022 At 1:09 pm NVMeoF: This all sounds interesting, and perhaps the concrete poured to make my mind has fully cured (my 1st storage task was to write SASI (then SCSI) disk & tape drivers for a proprietary OS)...But how does one take this baseline physical architecture up to the level of enterprise storage array services? (file server, snapshotting, backup, security/managed file access, control head failover, etc)? Hmmmm...Maybe I should just follow the old 1980's TV show "Soap" tag line "Confused? You won't be after the next episode of Soap". 3. Hans Henrik Happe April 20, 2022 At 2:01 pm This is really is a lot like SAS, but with Ethernet scale-out. The key thing is that you get the single drive small failure domain. At one end you could do a 1000+ drive archival system with a few servers that are fully redundant (server failover). At the other end you could put more umph into "controllers" by adding more servers and use storage accelerators like DPUs. Software defined systems based on plain servers have sizing issues. You spend to much on servers to maintain descent sized failure domains. Even if you decide to add a lot of space to each server, you need a lot of servers to keep the redundancy overhead down (erasure coding). Then you get into a multi PB system before it gets economical. Moving the software to the drives can solve this and given that this is already Ethernet it might not be that far out, but will have cost for on drive processing. With the good old dual path drive you can start with two servers (controllers) and a 10 drive RAID6 and now scale as far as Ethernet or any other NVMeoF network will take you. LEAVE A REPLY [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] Please enter your comment! [ ] Please enter your name here [ ] You have entered an incorrect email address! Please enter your email address here [ ] [ ]Save my name, email, and website in this browser for the next time I comment. [ ] [ ]Sign me up for the STH newsletter! [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] This site uses Akismet to reduce spam. Learn how your comment data is processed. [INS::INS] [INS::INS] Newsletter Get the best of STH delivered weekly to your inbox. We are going to curate a selection of the best posts from STH each week and deliver them directly to you. Your email address: [ ] [Sign up] By opting-in you agree to have us send you our newsletter. We are using a third party service to manage subscriptions so you can unsubscribe at any time. Leave this field empty if you're human: [ ] STH ABOUT US ServeTheHome is the IT professional's guide to servers, storage, networking, and high-end workstation hardware, plus great open source projects. Advertise on STH DISCLAIMERS: We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites. FOLLOW US (c) 2009-2022 Loyolan Ventures, LLC