https://old.reddit.com/r/databasedevelopment/comments/1afpezz/samsung_nvme_developers_ama/ jump to content my subreddits edit subscriptions * popular * -all * -random * -users | * AskReddit * -pics * -gaming * -mildlyinteresting * -worldnews * -movies * -explainlikeimfive * -todayilearned * -funny * -news * -dataisbeautiful * -OldSchoolCool * -nottheonion * -DIY * -videos * -TwoXChromosomes * -tifu * -books * -LifeProTips * -Showerthoughts * -science * -aww * -Music * -Jokes * -Futurology * -sports * -askscience * -space * -gadgets * -IAmA * -gifs * -history * -nosleep * -food * -GetMotivated * -UpliftingNews * -InternetIsBeautiful * -announcements * -philosophy * -creepy * -WritingPrompts * -Documentaries * -EarthPorn * -photoshopbattles * -listentothis * -blog more >> reddit.com databasedevelopment * comments Want to join? Log in or sign up in seconds.| * English [ ][] [ ]limit my search to r/databasedevelopment use the following search parameters to narrow your results: subreddit:subreddit find submissions in "subreddit" author:username find submissions by "username" site:example.com find submissions from "example.com" url:text search for "text" in url selftext:text search for "text" in self post contents self:yes (or self:no) include (or exclude) self posts nsfw:yes (or nsfw:no) include (or exclude) results marked as NSFW e.g. subreddit:aww site:imgur.com dog see the search faq for details. advanced search: by author, subreddit... this post was submitted on 31 Jan 2024 26 points (100% upvoted) shortlink: [https://redd.it/1afp] [ ][ ] [ ]remember mereset password login Submit a new link Submit a new text post Get an ad-free experience with special benefits, and directly support Reddit. get reddit premium databasedevelopment joinleave4,537 readers 104 users here now a community for 2 years MODERATORS * message the mods discussions in r/databasedevelopment <> X 26 * 59 comments Samsung NVMe developers AMA 6 Build a simple LSM-based KV storage engine in a week 1 DataFusion queries 3 * 2 comments Redis as write-behind cache on a Linux embedded device 5 [yKM63yjF] Adaptive _time compression -- A database hacker story 7 * 5 comments [xfAwl9ok] VART: A Persistent Data Structure For Snapshot Isolation 2 [FHhbZN26] Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1) 24 * 8 comments [568Zx2Vp] Rethinking my Rust LSM storage engine 1 * 4 comments Can't figure out the logic behind a code in SimpleDB 7 * 1 comment [FrVefAeL] Scalable OLTP in the Cloud: What's the BIG DEAL? Welcome to Reddit, the front page of the internet. Become a Redditor and join one of thousands of communities. x 25 26 27 Samsung NVMe developers AMA (self.databasedevelopment) submitted 4 hours ago by eatonphil Hey folks! I am very excited that Klaus Jensen (/u/KlausSamsung) and Simon Lund (/u/safl-os) from Samsung, have agreed to join /r/ databasedevelopment for an hour-long AMA here and now on all things NVMe. This is a unique chance to ask a group of NVMe experts all your disk/ NVMe questions. To pique your interest, take another look at these two papers: 1. What Modern NVMe Storage Can Do, And How To Exploit It: High-Performance I/O for High-Performance Storage Engines 2. I/O Interface Independence with xNVMe One suggestion: to even the playing field if you are comfortable, when you leave a question please share your name and company since you otherwise have the advantage over Simon and Klaus who have publicly come before us. * 59 comments * share * save * hide * report all 59 comments sorted by: best topnewcontroversialoldrandomq&alive (beta) [ ] Want to add to the discussion? Post a comment! Create an account [-]safl-os 9 points10 points11 points 4 hours ago* (1 child) Hello Reddit! I'm Simon A. F. Lund, a pragmatic programmer at heart and a computer scientist with a passion for pushing the boundaries of technology. My journey in this fascinating world of tech started when I was just 12, trading a year of newspaper deliveries for my very first computer. It came with Windows 95, but it wasn't long before I dived into the realms of FreeBSD, marking the beginning of what would become a lifelong passion for computing.Fast forward to today, and I've navigated through the intricate lanes of computer science, earning a PhD with my thesis titled "A High Performance Backend for Array-Oriented Programming on Next-Generation Processing Units." This work laid the groundwork for JIT-compilation and operator fusion techniques through experimental validation of prototypes in the Bohrium Runtime system, techniques that are now cornerstones in projects like Jax.But the journey didn't stop there. I pivoted from "compute" to "storage," where I've been instrumental in transforming the programmability of storage devices. This pivot started out with Open-Channel SSDs and evolved into NVMe in the form of ZNS and FDP. These new NVMe storage technologies increase the need for flexibility, efficiency, and control in programming storage devices. This has materialized in the xNVMe project, a beacon in the storage world that offers an efficient programming model and unifies various system interfaces, libraries, and APIs. It's all about "programming emerging storage devices," a concept I was thrilled to present at SYSTO22 with my paper "I/O interface independence with xNVMe." For those curious, a quick intro to xNVMe is available at http://xnvme.io , with intro slides here: https://xnvme.io/material/intro/ xnvme-intro.pdf.The latest feather in my cap is the integration of xNVMe and the io_uring_cmd "I/O Passthru" interface in SPDK, showcased in the SDC23 conference presentation titled "xNVMe and io_uring NVMe Passthrough - What does it Mean for the SPDK NVMe Driver?" (see here: https://storagedeveloper.org/events/agenda/ session/553) And most recently, I have contributed to the work accepted at FAST24, titled "I/O Passthru: Upstreaming a Flexible and Efficient I/O Path in Linux," which will be presented at FAST24 ( https://www.usenix.org/conference/fast24/presentation/joshi).I'm here on Reddit for an AMA, ready to dive into discussions about storage, programming, and perhaps a bit of nostalgia for the good old days of FreeBSD. So, go ahead - ask me anything! * permalink * embed * save * report * reply [-]safl-os 1 point2 points3 points 1 hour ago (0 children) I am signing off here as well, thanks for all the questions, it was really fun! Btw. in my daily function then I am available on the Discord Server named "Samsung Memory Open-Source" (invite link https://discord.com/ invite/XCbBX9DmKf ) feel free to join. I am going on paternity and back again in May, but my team is there to help out with additional questions on e.g. xNVMe * permalink * embed * save * parent * report * reply [-]KlausSamsung 6 points7 points8 points 4 hours ago (2 children) Hi r/databasedevelopment! I'm Klaus, your friendly neighborhood NVMe software engineer! Before getting into storage software stacks proper, I spent some time with the dark arts of High Performance Computing, and yes, I was that conservative UNIX sysop who probably avoided you - nothing personal, just a bit old school. I survived a stint in "IT", wrote my thesis about tape storage (pretty vintage, I know), and then dove headfirst into the exhilarating world of NVMe and, specifically, OpenChannel SSDs (SSDs are all sequential like tape anyway, so those four years wasn't a total waste). I then moved on to NVMe Zoned Namespaces and, recently, Flexible Data Placement, getting more involved with the NVMe community in general and contributing as a technical proposal co-author. I proudly co-maintain the NVMe emulation in QEMU and I'm the creator of libvfn, which fundamentally is a library for writing user space PCI drivers using VFIO/IOMMUFD. At Samsung, I lead a small, dedicated team, focusing on emerging technologies. When it comes to databases, I'm more of an enthusiast than an expert, but I do know my way around. I'm here for a great conversation on NVMe and emerging storage tech in general - And I'm super excited to dive in. So, let's go - Ask Me Anything! * permalink * embed * save * report * reply [-]KlausSamsung 1 point2 points3 points 1 hour ago (1 child) It's getting a little late here in Europe, so I'm gonna sign off for the night. I'll check back in the morning! :) Thanks for all your amazing questions - it's been a blast! * permalink * embed * save * parent * report * reply [-]eatonphil[S] 0 points1 point2 points 1 hour ago (0 children) Thank you for your time Klaus!! * permalink * embed * save * parent * report * reply [-]gabrielhaasdb 4 points5 points6 points 3 hours ago (2 children) Hi Klaus and Simon! I'm Gabriel, author of the "What Modern NVMe Storage Can Do... " Paper. I'm curious about what the future holds for io_uring_cmd/I/ O-Passthru. I understand its usefulness in sending arbitrary NVMe commands to the SSD, but is it also supposed to (drastically) increase performance by bypassing the FS/block device abstractions? Like getting close to SPDK efficiency? I tried it out a while ago using xNVME but didn't really see any benefits. Also looking forward to the FAST24 paper! * permalink * embed * save * report * reply [-]safl-os 2 points3 points4 points 3 hours ago (1 child) Hi u/gabrielhaasdb ! I read your paper, excellent work, we should talk some more / have a deep-dive on what you saw when using xNVMe in Leanstore to unlock the missing performance. Because, yes, there is a benefit over regular io_uring there is an evaluation / comparison in the FAST24 paper, however, for immediate numbers then have a look at the SDC23 presentation: https://www.youtube.com/watch?v=Y7A3dPpdjNs Numbers from the SDC23 presentation (out-of-context); io_uring 4.1M IOPS, io_uring_cmd 4.86M IOPS, SPDK 8.08M IOPS. Now, this might seem like a huge gap, however, when using e.g. libaio then it "flatlines" at < 2M IOPS. Thus, io_uring provides better scalability, and io_uring_cmd even more so as it has less code on its path. Now, the scalability flatline for libaio, this same issue will come up when using io_uring and io_uring_cmd, if the NVMe-driver is not setup with poll-queues, and not driven with the optimal batch submission/completion sizes, etc. Thus, when not instrumented optimally, then you will see the same flat-line as the one seen with libaio caused by being bottle-necked by kernel/user-space context switching and interrupt processing. Having said this, then work is continuing on improving the efficiency of general io_uring and io_uring_cmd / I/O Passthru, so, I would expect the gap to narrow. * permalink * embed * save * parent * report * reply [-]gabrielhaasdb 2 points3 points4 points 2 hours ago (0 children) Thank you, Simon! I looked at the SDC23 numbers, it's interesting that the benefit is when batching submit/completions. I'll do some benchmarking on our new PCIe 5.0 SSDs in the next weeks and will have a look at it. I'll reach out to you! * permalink * embed * save * parent * report * reply [-]_shadowbannedagain 2 points3 points4 points 3 hours ago (4 children) Hello, Jaromir from QuestDB here. I have a somewhat open-ended and not really HW-specific question, but that's something I was thinking about recently: io_uring seems to be the go-to Linux interface for data-intense applications these days. However, there have been security concerns recently. It's even been disabled by some cloud providers. Do you think it's just a function of maturity and it'll get better over time or it's more fundamental - io_uring is rather different than most of "normal" syscalls? * permalink * embed * save * report * reply [-]KlausSamsung 4 points5 points6 points 3 hours ago (2 children) I'm aware that `io_uring` has had some security issues, but it is getting a lot of love on that front from the kernel community these days. It is definitely not fundamentally flawed. It's exceptional and it *is* maturing as we speak. * permalink * embed * save * parent * report * reply [-]_shadowbannedagain 2 points3 points4 points 3 hours ago (1 child) thanks! I have one more question: Do you foresee any trend in storage performance/behaviour which most application developers can't see (yet) and this trend will influence how we design applications and API in the years to come? * permalink * embed * save * parent * report * reply [-]KlausSamsung 4 points5 points6 points 2 hours ago (0 children) Yes, I believe that Data Placement technologies are here to make a difference. There is wide support for zoned namespaces within the Linux kernel block stack and we are seeing applications like Meta's CacheLib merging support for utilizing Flexible Data Placement. These technologies will most likely impact future high-performance application design for the better. * permalink * embed * save * parent * report * reply [-]docsavage 3 points4 points5 points 3 hours ago (1 child) Hi, I was wondering about the current state of the key-value API and the possibility of wider availability in products? I'm Bill Katz working at HHMI Janelia Research Campus. * permalink * embed * save * report * reply [-]safl-os 4 points5 points6 points 3 hours ago (0 children) a Hi u/docsavage, great question! From an NVMe perspective then it is standardized and the specification made publicly available here: https://nvmexpress.org/ wp-content/uploads/ NVM-Express-Key-Value-Command-Set-Specification-1.0d-2024.01.03-Ratified.pdf Thus, it is well-defined how KV-Commands are constructed and theory-of-operations on the KV-Command set. From an API perspective then the xNVMe project provides a C API, with bindings from Python and Rust, you can have a look here: * C API: https://xnvme.io/docs/latest/capis/xnvme_kvs.html * Usage example in a kvs cli-tool here: https://github.com/OpenMPDK /xNVMe/blob/main/tools/kvs.c * With tests written using the Python bindings here: https:// github.com/OpenMPDK/xNVMe/blob/main/python/tests/ test_key_value.py I cannot speak much to product availability, so you might wonder how the above tests are run, and the answer to that is using device emulation via qemu. If you take xNVMe for a spin then you can fire up a qemu instance with KV device emulation by running the following: # Grab the xNVMe source git clone https://github.com/OpenMPDK/xNVMe.git xnvme cd xnvme git checkout next # Use the script that matches you Linux distribution to install dependencies ./toolbox/pkgs/ # Setup development environment in a qemu guest / vm make cijoe make cijoe-guest-setup-xnvme-using-git With the above incantation, then you have a qemu-guest / virtual machine that you can SSH into with ``ssh -p 4200 root@localhost`` and the machine will present NVMe devices with namespaces with NVM, ZNS, KV command-set support. * permalink * embed * save * parent * report * reply [-]micvbang 3 points4 points5 points 3 hours ago (1 child) Thanks for doing this AMA! I was wondering - which developments in NVMe landscape are you most looking forward to? Both from a community wide and personal point of view. Do you see any technological, political, or other hurdles to get there? * permalink * embed * save * report * reply [-]KlausSamsung 2 points3 points4 points 2 hours ago (0 children) Everything related to 'Data Placement' is pretty exciting. I think it is the best thing that could happen to NVMe. I'm also pretty excited about the possibilities of Key Per I/O (basically, in the extreme, the ability to use a distinct encryption key per I/O command) - it should be pretty interesting to databases! There are also a bunch of exciting stuff going on behind the NDA curtain within the NVMe Technical Working Groups of course. But, you know, NDA... * permalink * embed * save * parent * report * reply [-]smasher164 3 points4 points5 points 3 hours ago (1 child) Given NVME's high throughput in random I/O, is it still worth it to maximize sequential I/O in the architecture of a database? I'm assuming for the moment that existing kernel interfaces and compatibility with old hardware interfaces isn't a concern. * permalink * embed * save * report * reply [-]KlausSamsung 2 points3 points4 points 2 hours ago (0 children) Sort of. If you write sequentially, you make the FTL's life easier. However, a better (and more predictable) approach is to optimize by exploiting explicit data placement (like, ZNS or FDP). * permalink * embed * save * parent * report * reply [-]linearizable 2 points3 points4 points 3 hours ago (0 children) Where is the bottleneck in modern NVMe SSDs? Are the internal physical reads/writes drive the slowest part? Is the bus the slowest? Is it the decoding and dispatching between the two? If you could take an ssd-wide flamegraph during saturation, where would the time be going? * permalink * embed * save * report * reply [-]linearizable 3 points4 points5 points 3 hours ago* (2 children) When new features in storage are being worked on which involve exposing new functionality to userland, e.g. a new addition to the NVMe protocol meaning there's a new API for interacting with the drive, how is the process for actually getting that into something that can be invoked in linux(/windows/mac) userland? (I'm looking at you, difficult to invoke and poorly supported fused compare-and-write.) * permalink * embed * save * report * reply [-]KlausSamsung 2 points3 points4 points 3 hours ago (1 child) Ha. Yeah, 'fused' commands comes up on the linux-nvme mailing list now and then. The quick answer is, just use a user space driver if you need custom functionality that is not covered or possible with the Linux kernel driver. But if you actually need the fs and/or block layer, then you're out of luck. However, the introduction of `io_uring_cmd` has changed this landscape quite a bit. You can now send NVMe commands directly to the drive with `io_uring`. However, you still have to work *with* the driver, not against it. `io_uring_cmd` is what allows xNVMe to work with key/value drives without relying on a user space driver. I'll let Simon do a follow-up and correct me here if needed. He's the expert on this ;) * permalink * embed * save * parent * report * reply [-]safl-os 2 points3 points4 points 2 hours ago (0 children) Adding a couple of details on this. Specifically on "how is the process for actually getting that into something that can be invoked in linux(/windows/mac) userland?". Now, the really excellent thing is that today a project such as xNVMe can implement this in a library, constructing the 64bytes defining with NVMe command with opcode etc. then xNVMe passes this to its collection of backend implementations to transport that command to the device along with the payloads. The code looks something like this: int xnvme_nvm_write_zeroes(struct xnvme_cmd_ctx *ctx, uint32_t nsid, uint64_t slba, uint16_t nlb) { ctx->cmd.common.opcode = XNVME_SPEC_NVM_OPC_WRITE_ZEROES; ctx->cmd.common.nsid = nsid; ctx->cmd.write_zeroes.slba = slba; ctx->cmd.write_zeroes.nlb = nlb; return xnvme_cmd_pass(ctx, NULL, 0, NULL, 0); } Thus, the quick answer is; someone implements the command-construction as defined by the NVMe specification similar to the above and sends a pull-request to e.g. xNVMe :)Now, the above is one way of defining commands in a library, what xNVMe does then is transporting it to the device via one of the following I/O paths: * The Linux Kernel driver ioctl() interface * The io_uring_cmd / I/O Passthru interface * The SPDK user-space NVMe driver * The libvfn user-space NVMe driver * The FreeBSD NVMe driver ioctl() interface All of these provide "passthru" interfaces, which enables a library such as xNVMe to handle the command construction and send it down through the I/O path that best serves the application / use-case. * permalink * embed * save * parent * report * reply [-]Legal_Artist 2 points3 points4 points 4 hours ago* (3 children) Hello, I am Dongjoo Seo who 3rd yr phd student from UCI. What do you think about the polling as a completion method even for sync I/O operation? * permalink * embed * save * report * reply [-]safl-os 2 points3 points4 points 3 hours ago (2 children) sync Hi u/Legal_Artist , thanks for the question, can you elaborate a bit? Usually, when referring to sync I/O that would be blocking system calls or APIs where the "caller" waits for the function to do "its thing", where "its thing" behind the scenes can be something like waiting for interrupts/signals, processing a completion-queue, waiting for a callback-function to trigger, etc. thus, there is not much polling to do from perspective of the "caller". * permalink * embed * save * parent * report * reply [-]Legal_Artist 1 point2 points3 points 3 hours ago (1 child) Sure. what I mean is that interrupt is already taken 1~2us for their operation overhead for NVME operation. this should be fine when we are sending batching I/O or async I/O to Disk because it is only has a few portions of total overhead. But, in sync I/O request when the device is fast enough like 8~11us level, interrupt overhead taken increases the latency over 10%. So, my question is can we somehow delete the interrupt overhead even for the sync-based I/O operation? * permalink * embed * save * parent * report * reply [-]KlausSamsung 1 point2 points3 points 2 hours ago (0 children) Yes. Even though your API behaves like it's synchronous, you can create completion queues that are not interrupt-driven. In that case, the kernel will set a side resources to poll it instead. * permalink * embed * save * parent * report * reply [-]pancreas101 2 points3 points4 points 3 hours ago (2 children) CJ, new grad Thanks for doing the AMA!! (1) Top book recs? Not necessarily technical, but really anything that's had an outsized impact on you, or is just a fun read! (2) A little cliche, but any advice you wish you'd internalized as a younger engineer? * permalink * embed * save * report * reply [-]KlausSamsung 2 points3 points4 points 3 hours ago (0 children) 1) The C Programming Language, Surely You're Joking Mr. Feynman! and Masters of Doom. 2) Rob Pike's "5 Rules of Programming" comes to mind. * permalink * embed * save * parent * report * reply [-]safl-os 1 point2 points3 points 1 hour ago* (0 children) Hi u/pancreas101 ! 1. Some of my favorites are * The C Programming Language by Brian Kernighan and Dennis Ritchie * The Mythical Man-Month by Fred Brooks + Especially the "No silver-bullet" and "The Tar Pit" * The Pragmatic Programmer by Andy Hunt and Dave Thomas + Also the "new" version, the book has aged well * The Cathedral and the Bazaar Essay by Eric S. Raymond * The Jargon File, also by ESR + http://www.catb.org/jargon/ * Programming, MotherfuckerDo you speak it? + http://programming-motherfucker.com/ + This one is mostly just for fun... but the first time I read it... I found it **very** funny :) * permalink * embed * save * parent * report * reply [-]Legal_Artist 2 points3 points4 points 3 hours ago (0 children) For ZNS and FDP, Do you know any plan for the new file system that can get the advantage of them? F2FS, BTRFS, and ZoneFS kinds of things exist but eventually, we are facing lots of issues like garbage collection in user space or utilization. only developed app for ZNS or FDP is solutions for it? * permalink * embed * save * report * reply [-]midearth_citizen 2 points3 points4 points 3 hours ago (1 child) Hey folks. I'm Seyed Mohammad. I'm currently a freelance developer. In my previous roles I worked on implementing an IAAS system alongisde Gluster for distributed storage. Thank you for this opportunity. I'm interested in NVMe+io_uring implications for virtualized workloads. Given your extensive experience, especially with NVMe alongside QEMU, how do you foresee NVMe's role evolving in the context of virtualization? Additionally, are there specific challenges or considerations when optimizing NVMe usage for virtualized environments, especially when leveraging newer I/O paradigms like io_uring. Thanks * permalink * embed * save * report * reply [-]KlausSamsung 1 point2 points3 points 2 hours ago (0 children) First, I'm gonna assume that you are referring to the prospect of having a virtualized NVMe device as your root disk on your VM instead of a virtio-based one. Currently, NVMe emulation in QEMU is slow. We have a patchset that brings NVMe emulation up along side virtio, but it still needs a little more testing. We had an Google Summer of Code intern, Jinhao, that did amazing work on this. But the question remains. Why would you want to do that? The only reason to use an emulated nvme device instead of a virtio device, in production, would be because your guest OS for some reason had crappy virtio block drivers, but great nvme drivers. That is unlikely to be the case. For now, the primary use case for QEMU NVMe emulation is to enable developers to experiment. Now, assuming the above was not what you meant. Today, it is already pretty normal to be able to hot-add an *actual* NVMe device into your virtual machine using PCI passthrough. That gives you access to all of those millions of IOPS that you crave ;) And just like on bare metal, you need something like xnvme or raw io_uring to actually unlock that performance. * permalink * embed * save * parent * report * reply [-]shikhar-bandar 1 point2 points3 points 3 hours ago (2 children) Can we expect ZNS SSDs to be available in the cloud? Do you think hyperscalers will just use them to divvy up SSDs between VMs? * permalink * embed * save * report * reply [-]linearizable 2 points3 points4 points 3 hours ago (1 child) To tack onto this thread, it seems like a lot of improvements made in the hardware space do not seem to not trickle down to consumer drives well, nor drives available in the cloud. For example, I'm not aware of any possible way to purchase or cloud rent an SMR HDD. Why is this, and which of the current and future hardware improvements would you expect to be more available than just "enterprise-only" drives. (E.g. ZNS mentioned here, but also: KV interface, >4KB atomic writes, computational storage?) * permalink * embed * save * parent * report * reply [-]riksi 1 point2 points3 points 3 hours ago (0 children) I'm not aware of any possible way to purchase or cloud rent an SMR HDD. Why is this So you don't build your own S3 on top of it. Same reason the clouds have expensive bandwidth. * permalink * embed * save * parent * report * reply [-]linearizable 1 point2 points3 points 3 hours ago (8 children) Papers discussing write optimized storage engines compare write amplification measured as writes submitted to the drive, which discounts the existence of the FTL and how workloads may favorably or unfavorably interact with it. How would you recommend measuring full end-to-end write amplification? Is just measuring throughput over an extended span of time an actually sufficient proxy? * permalink * embed * save * report * reply [-]KlausSamsung 2 points3 points4 points 3 hours ago (0 children) If you have an SSD on hand that implements the OCP Datacenter NVMe SSD Specification, then there is a standardized log page (the "SMART / Health Information Extended") that has a "Physical Media Units Written" field. You can use that to calculate WAF. Some consumer drives also implement that log page in some form (as log page identifier C0h). `nvme-cli` has support for a bunch of them for various vendors. * permalink * embed * save * parent * report * reply [-]linearizable 1 point2 points3 points 3 hours ago (5 children) And as a sort of part 2 to this, FTLs are continuously improving/ changing and no vendor seems to publicly talk about their FTL or how to optimize for it? If I ran my b-tree against an FTL simulator, would that be more of a helpful simulation like using cachegrind, or would it be more like optimizing a program for x86 and then running on ARM? * permalink * embed * save * parent * report * reply [-]KlausSamsung 1 point2 points3 points 3 hours ago (4 children) Yeah, FTL logic and garbage collection algorithms are closely guarded trade secrets. I'm not super much into simulation (I'm mostly doing emulation, which is just high-level modeling for prototyping functionality), but as far as I know, simulators like MQSim gives a pretty decent result wrt. latencies, but their accuracy on WAF could be off by quite a bit. * permalink * embed * save * parent * report * reply [-]eatonphil[S] 0 points1 point2 points 3 hours ago (3 children) > but their accuracy on WAF could be off by quite a bit. (Anyone in the know:) What does WAF mean here? * permalink * embed * save * parent * report * reply [-]katnegermis 2 points3 points4 points 3 hours ago (2 children) Write amplification; the total number of bytes written to the underlying storage, including internal SSD stuff, not just the bytes you think wrote to it from the OS :) * permalink * embed * save * parent * report * reply [-]KlausSamsung 2 points3 points4 points 2 hours ago (0 children) Right on. Rather than going deeper into it, I'm just gonna drop a link to Everything I know about SSDs, which is a great short resource on understanding how SSDs *really* work. * permalink * embed * save * parent * report * reply [-]eatonphil[S] 0 points1 point2 points 3 hours ago (0 children) Thank you! * permalink * embed * save * parent * report * reply [-]gabrielhaasdb 2 points3 points4 points 3 hours ago* (0 children) Look for SSDs that support the OCP specification, they will report physical media reads/writes. This should be the actual writes that are done on flash, so including FTL stuff. We have some new intel and kioxia enterpris SSDs that support it, don't know about Samsung. nvme-cli command: nvme ocp smart-add-log /dev/nvme.. * permalink * embed * save * parent * report * reply [-]govi20 1 point2 points3 points 3 hours ago (1 child) Any advice for someone who wants to switch from web dev to NVMs (or low-level development)? * permalink * embed * save * report * reply [-]safl-os 1 point2 points3 points 2 hours ago (0 children) I would say, jump right in, get your feet wet and your hands dirty :) The xNVMe project has a guide on setting up a Linux development environment: https://xnvme.io/docs/latest/tutorial/devs/index.html With this you have a test-bed where you can poke around with virtual NVMe devices and experiment with how they behave and interact with them using cli-tools, write C programs using the xNVMe C APIs and interactively using the Python APIs. Or using the Rust bindings if that is more your taste. That might be one way to get into it :) Another thought is that, if the C code might be a bit too rough coming from web-development, then Linux kernel has started to do a lot of work around Rust. Specifically then our colleague Andreas Hindborg has done some exciting work on the Block Layer and the NVMe driver: * https://www.youtube.com/watch?v=lI9DLvxUA4Q * https://www.youtube.com/watch?v=BwywU1MqW38 * https://www.youtube.com/watch?v=ubohmQSTeBY This might be nicer way to get into low-level development via the support of the Rust toolchain. Starting out with some embedded development using Rust on micro-controllers etc. and them switching over to see how Rust in the Linux kernel looks like. * permalink * embed * save * parent * report * reply [-]Healthy-Seesaw-9700 1 point2 points3 points 3 hours ago (1 child) Hi all, from Belgium Simon, you mentioned "io_uring NVMe Passthrough" and being a FreeBSD user. Is there a platform agnostic equivalent? (as io_uring is Linux based). Where do you see the industry headed as targetting purely io_uring in several products (scylladb, dragonflydb etc. etc.) will limit deployment choices. * permalink * embed * save * report * reply [-]safl-os 2 points3 points4 points 2 hours ago (0 children) Hi u/Healthy-Seesaw-9700 ! From my perspective then using a storage-abstraction-layer such as xNVMe would prevent "vendor-lock-in" to Linux/io_uring. xNVMe would be the closest to a platform agnostic equivalent, as the xNVMe APIs have implementations utilizing the FreeBSD NVMe driver ioctl() interface, as well as support for the "traditional" I/O operations via POSIX aio, and even Windows IOCP. On the FreeBSD side, then last year xNVMe received a contribution for kqueue-based aio (https:// github.com/OpenMPDK/xNVMe/pull/286) and this year xNVMe made it into the FreeBSD ports tree: https://cgit.freebsd.org/ports/tree/sysutils/ xnvme And something currently under active development is automated performance evaluation of xNVMe on Linux, FreeBSD, and Windows. This will help shed some light on capabilities of the different platforms. I am really looking forward to the benchmark results comparing the interfaces on the different platforms with baselines of reference implementations to the same interfaces utilized via xNVMe. In other words, I would really not like to see the industry heading down a road of less freedom, rather I would like to contribute to the free mobility of storage applications. * permalink * embed * save * parent * report * reply [-]linearizable 1 point2 points3 points 3 hours ago* (1 child) xNVMe is a great abstraction layer over operating directly on storage, and I appreciate the backend to allow modification of regular files. When I think about shipping a database built using xNVMe, I'm feeling concerned that for developers using the database to build applications, having to statically allocate out a maximum dataset size feels unkind, and they're very unlikely to have a spare attached SSD instead. Having a storage implementation that can work on a growable/shrinkable file would be the ideal there, which I think would leave one currently writing an abstraction layer over xNVMe as well. Is there any philosophical resistance to extending the IO support (xnvme_spec_fs_opcs?) such that e.g. xnvme_be_linux_async_liburing could use a growable or shrinkable storage, or is it mostly just an issue of someone volunteering to do the work? * permalink * embed * save * report * reply [-]safl-os 1 point2 points3 points 2 hours ago (0 children) Hi u/linearizable, wow thanks and great question! The focus on the xNVMe APIs have been to enable flexibility / control of NVMe features and support and the file APIs have received less attention. Suggestions to how to improve these, and especially suggestions guided by a usage-case such as what you describe would be very welcome. There are philosophical/design resistance towards such efforts, the only barrier here is someone volunteering to do the work and showcasing the benefits :) In this area, then the work that u/gabrielhaasdb has done with Leanstore might also be worth exploring further or in a different direction, e.g. what are the ideal storage-primitives to build on top of the unified API provided by xNVMe. There is definitely a lot of exciting exploration to do here :) * permalink * embed * save * parent * report * reply [-]photoszzt 1 point2 points3 points 2 hours ago (1 child) Hi, I might be late to the party. There's a new device called memory semantic SSD(CMM-H) from Samsung that can operate with CXL protocol. What do people think about the protocol compared to nvme? I saw the webpage talks about accelerating AI workload? Is there any real win on the actual workload? Is there any insight on using both protocols together? * permalink * embed * save * report * reply [-]KlausSamsung 1 point2 points3 points 2 hours ago (0 children) I'm not an expert on CXL or that specific product, but as far as I know, the trick is that, being exposed as a CXL device, it is byte-addressable (compared to NVMe that is operating on blocks of, say, 512 or 4096 bytes). It basically gives you an SSD sized chunk of extra cache coherent non-volatile memory that you can modify in-place instead of moving it from the SSD into RAM first. * permalink * embed * save * parent * report * reply [-]snabx 1 point2 points3 points 1 hour ago (1 child) This is probably the most exciting post I've seen so far since I joined this sub. I don't have any questions except maybe some advice for getting closer to working with databases/storage coming from data engineering job. * permalink * embed * save * report * reply [-]KlausSamsung 1 point2 points3 points 1 hour ago* (0 children) Just Do It! The storage community is here to help and answer your questions! And see u/safl-os's previous answer! Edit, as the maintainer of the NVMe emulation in QEMU, I'm a bit biased here, but I'll highlight that the ability to bring up an emulated test bed that you can just *trash* by sending random commands and modify to support your crazy ideas is just amazing. * permalink * embed * save * parent * report * reply [-]CompSci_01 -1 points0 points1 point 4 hours ago (1 child) meeting link? * permalink * embed * save * report * reply [-]eatonphil[S] 1 point2 points3 points 4 hours ago (0 children) You're in it already. :) * permalink * embed * save * parent * report * reply [-]CompSci_01 0 points1 point2 points 4 hours ago* (1 child) for NVMe can we have a command that can send and receive data? * permalink * embed * save * report * reply [-]KlausSamsung 1 point2 points3 points 3 hours ago (0 children) Can you elaborate a bit? NVMe Read and Write pretty much does that :P If you are talking about network stuff, then you wanna get into NVMe over Fabrics. * permalink * embed * save * parent * report * reply [-]kkgmgfn 0 points1 point2 points 3 hours ago (0 children) I used to work in Samsung as Dev way back in 2013. Any openings now? * permalink * embed * save * report * reply * about * blog * about * advertising * careers * help * site rules * Reddit help center * reddiquette * mod guidelines * contact us * apps & tools * Reddit for iPhone * Reddit for Android * mobile website * <3 * reddit premium Use of this site constitutes acceptance of our User Agreement and Privacy Policy. (c) 2024 reddit inc. All rights reserved. REDDIT and the ALIEN Logo are registered trademarks of reddit inc. [pixel] p Rendered by PID 47 on reddit-service-r2-loggedout-8d596488f-frhsw at 2024-01-31 23:02:05.541624+00:00 running 1127052 country code: US.