[HN Gopher] Fire-Flyer File System from DeepSeek
___________________________________________________________________
Fire-Flyer File System from DeepSeek
Author : wenyuanyu
Score : 326 points
Date : 2025-02-28 01:26 UTC (21 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| do_not_redeem wrote:
| Can someone convince me this isn't NIH syndrome? Why would you
| use this instead of SeaweedFS, Ceph, or MinIO?
| startupsfail wrote:
| It's not. When you are a high frequency trader and you've
| mastered RDMA, everything around you looks slow. You are
| thinking in terms of 20 nanoseconds intervals, while everyone
| around still thinks that serving a query under a millisecond is
| fast.
| rfoo wrote:
| Huh? What kind of RDMA has a completion latency of 20
| nanoseconds? It's more like 5 microseconds.
|
| I agree that a lot of "modern" storage stack is way too slow
| though, tried to find a replication-first object storage for
| crazy-fast random read in small number of objects last year
| and found none.
| tucnak wrote:
| Completion latency is one thing, bandwidth would be
| another. There's apparently a whole world of Alveo
| SmartNIC's and related FPGA platforms, and it can totally
| get in nanosecond range for whatever nails that may fit the
| compute-in-network hammer, even if bound by latency of the
| consuming system / RDMA interface. Also:
| https://github.com/corundum/corundum is really popular with
| the Chinese!
| startupsfail wrote:
| I was talking about, thinking in terms of 20 nanoseconds
| intervals, rather than completing a request in 20
| nanoseconds. To get 1 microsecond wire-to-wire latency you
| do need to count your nanoseconds.
|
| Why this number - this is because it's roughly the time it
| takes to read 64 bytes from L3 cache. And NICs tend to be
| able to push data into L3 (or equivalents).
|
| Current state of the art - look up nanoPU, from Stanford.
| Wire-to-wire under 100ns is not impossible, but this would
| normally assume pre-cooked packet, selected from a number
| of packets (which is not an unusual scenario in HFT).
| rfoo wrote:
| Ah, makes sense. Sadly RDMA isn't that fast for now, or
| at least commercial RNICs/switches don't :( Once you left
| your host in data center network, everything counts in
| microseconds.
| ein0p wrote:
| Seems like Ceph is considerably lower in throughput:
| https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/ A
| serious concern when saving hundreds of terabytes of weights
| and optimizer states every now and again, or loading large
| precomputed prefix KV-caches. Minio seems to be slower still.
| IDK about SeaweedFS - they don't mention performance in their
| selling points at all.
| do_not_redeem wrote:
| It's quite funny that I got two opposite answers right away:
| you say it's to improve throughput, and sibling says it's to
| improve latency, and as we know throughput and latency trade
| off against each other. I'm inclined to agree it's more
| likely they're prioritizing throughput, since their readme
| charts throughput but not latency. But OTOH, the project
| looks like it requires RDMA. I wonder if the authors have
| written about their motivations and the tradeoffs they made,
| so we don't have to speculate.
|
| EDIT: Their blog post answered all my questions and more.
| https://www.high-flyer.cn/blog/3fs/
| ein0p wrote:
| Because the two are interconnected and aren't in conflict
| with each other. You not only want high throughput - that
| by itself would be quite limiting. You want it along with
| low latency as well, or else it's very easy to end up in
| the situation where your throughput is effectively zero if
| the access pattern is "bad".
| mgerdts wrote:
| > The final aggregate read throughput reached approximately 6.6
| TiB/s with background traffic from training jobs.
|
| The Ceph team has been working on Crimson for years to get past
| performance bottlenecks inherent to the HDD-based design. I'm
| having troubles finding any ceph benchmark results that show
| any close to 100 GB/s.
| pat2man wrote:
| Seems easy to find: https://ceph.io/en/news/blog/2024/ceph-a-
| journey-to-1tibps/
| menaerus wrote:
| 3FS: 180 nodes, 2x200Gbps InfiniBand and 16x 14TiB NVMe
| SSDs per node, ~500 clients, 6.6 TiB/s of read throughput
| with training jobs workload
|
| Ceph: 68 nodes, 2x100Gbps Mellanox and 10x 14TiB NVMe SSDs
| per node, 504 clients, 1TiB/s of FIO random read workload
| nivertech wrote:
| I'd argue that they don't need a filesystem or an object
| storage, they need a purpose-built data serving layer
| optimized for their usecase.
| cttet wrote:
| If NIH syndrome boosts morale of the team, it should be helpful
| on overall team progress though.
| jpgvm wrote:
| None of those are close to fast enough.
|
| The only competitors in the parallel FS space that are useful
| for this are Lustre and Weka.
|
| Otherwise if you don't need a single namespace a bunch of fat
| AF NFSv4 servers w/NFS over RDMA will also get you to 6TiB/s.
|
| The "surefire" way though is still Lustre, it's the big daddy
| of distributed parallel filesystems still but it's an absolute
| beast to setup and operate.
| doomleika wrote:
| Software tech in China is a different landscape. It's really
| common to reinvent the wheel in China. Almost every big name
| (Bytedance, Meituan, etc) have their own of everything with
| both office political and in house need reasons.
|
| The thing is those stuff are so prevalent those in house tech
| have reach the point they are competitive. This doubles for
| quant firm like DeepSeek.
| whalesalad wrote:
| Sometimes you must succumb to NIH. How do you think all those
| tools you mentioned got produced?
| brcmthrowaway wrote:
| What does Anthropic use?
| pepsi-not-coke wrote:
| I love it. AWS EFS costs too much. The open source solutions are
| clunky. I'm hoping DS applied their ingenuity to this one, too.
| Can't wait to trial it.
| tetron wrote:
| Was curious how they get such performance with a FUSE based
| design. It seems that they sort of cheat, FUSE is used to manage
| metadata but to get high performance you have to link in the C++
| client library and do all your reads and writes through that. So
| it isn't general purpose, you have to modify your application to
| take advantage of it. Still, that's a clever trick, and makes me
| wonder if there's a LD_PRELOAD strategy that could generalize.
| grohan wrote:
| They appear to have Python bindings which seems reasonable from
| an API / usability perspective? https://github.com/deepseek-
| ai/smallpond
|
| In terms of fast FUSE - also my first question, appears to
| be`io_uring` + FUSE :)
|
| https://github.com/deepseek-ai/3FS/blob/main/src/lib/api/Usr...
| amelius wrote:
| Why is FUSE that much slower than providing your own read/write
| functions? I get that it has to go through the kernel, but the
| operations are on entire blocks and network should be the
| bottleneck by far (and disk/main memory should be a bottleneck
| if the data is local).
| vlovich123 wrote:
| You have to bounce through the kernel back out to use space.
| The number of syscalls is quite high. In many cases this is
| mitigated somewhat by the page cache making reads cheaper,
| but that's explicitly an anti design here.
|
| I believe there's work to minimize this using io_uring so
| that you can talk to the fuse driver without the kernel being
| in the middle, but that work isn't ready last time I checked.
|
| For what it's worth at Palm we had a similar problem because
| our applications were stored compressed but exposed through
| fuse uncompressed, instead of O_DIRECT I just did an fadvise
| to dump the cache after a read. Not as high throughput but
| the least risky change to get the same effect.
| sweettea wrote:
| Fuse over io_uring has just been merged:
| https://www.phoronix.com/news/Linux-6.14-FUSE
|
| So has uncached buffered IO:
| https://www.phoronix.com/news/Uncached-Buffered-IO-
| Linux-6.1...
|
| 6.14 is an exciting kernel!
| ammo1662 wrote:
| For those who are interested, the design was originally published
| here:
|
| (Chinese) https://www.high-flyer.cn/blog/3fs/
|
| This file system has been developed and utilized by them for
| several years .
|
| Compared to the traditional file systems, it is more focused on
| model training that contains a lot of random reads. Read cache
| and prefetching are useless in this case. Therefore, they
| designed the file system without those features to improve the
| performance.
|
| I google translated some key parts here:
|
| 3FS is a special file system because it is almost only used in
| the scenario of batch reading sample data in the computing node
| during AI training, and accelerates model training through high-
| speed computing and storage interaction. This is a large-scale
| random reading task, and the read data will not be used again in
| a short time, so we cannot use the most important tool "read
| cache" to optimize file reading, and even advance reading is
| useless. Therefore, the implementation of 3FS is also quite
| different from other file systems.
|
| Specifically, as shown in the figure above, 3FS uses the Linux-
| based AIO and io_uring interfaces to complete sample reading,
| because in the 3FS scenario, File Cache has no effect at all, but
| will consume system memory in a way that is difficult for users
| to control, affecting the operation of subsequent tasks, so we
| turned off File Cache and only used Direct I/O mode to read data.
| But it should be noted that when reading in this way, the buffer
| pointer, offset and length all need to be aligned. If the user is
| allowed to do this alignment, additional memory copies will be
| generated, so we have done the alignment inside the file system,
| which not only optimizes performance but also facilitates users.
| rvba wrote:
| Why is that a random read? Also is it truely random, or from
| seed? But if prng then they could cache right?
| kilburn wrote:
| Random is prng. They still cannot cache though because they
| do many reading "passes" through the same data.
|
| If build a cache that gets hits on the first pass, then it
| won't work for the second and later passes.
| vlovich123 wrote:
| I hope they chose a multiple of 4096 for the alignment to
| minimize flash read amplification. QLC drives even use 16kib
| pages.
| dekhn wrote:
| How critical is random reading of training data when assembling
| batches?
|
| Put another way: in my experience, supporting fast random reads
| is a challenging problem, while supporting high sequential
| reads is fairly straightforward. When is random access to a
| training set absolutely necessary for training a model?
| c4wrd wrote:
| Imagine you're studying for a test where you are given an
| image and need to answer the correct class. To prepare,
| you're given a deck of flashcards with an image on the front
| and the class on the back.
|
| (Random) You shuffle the deck every time you go through it.
| You're forced to learn the images and their classifications
| without relying on any specific sequence, as the data has no
| signal from sequence order.
|
| (Fixed order) Every time you go through the deck, the images
| appear in the exact same order. Over time you may start to
| unconsciously memorize the sequence of flashcards, rather
| than the actual classification of each image.
|
| When it comes to actually training a model, if the batches
| are sampled sequentially from a dataset, it risks learning
| from correlations caused by the sequencing of the data,
| resulting in poor generalization. In contrast, when you
| sample the batches randomly, the model is biased and
| encouraged to learn features from the data itself rather than
| from any signals that arise from artifacts of the ordering.
| arjvik wrote:
| On an SSD, random and sequential reads have nearly the exact
| same performance. Even on large arrays of spinning rust this
| is essentially true.
| budududuroiu wrote:
| Does anyone know if there's a benefit to porting this to an
| orchestrator like K8s, maybe overkill for training but the
| KVCache might be useful when having multiple replicas for
| inference?
| jeffbee wrote:
| Interesting that their GraySort result is CPU bound while they
| are using 3x more CPUs than the record holder from ten years ago.
| sitkack wrote:
| How can you determine that it CPU bound from the attached
| charts?
| jeffbee wrote:
| Because it hits a read peak in the first wave and never hits
| it again.
| sitkack wrote:
| Could be many other reasons, giving it more CPU won't
| necessarily increase the read rate.
| bee_rider wrote:
| They sure are productive.
|
| What are we going to see tomorrow? DeepSeek OS or something?
| logicallee wrote:
| >They sure are productive.
|
| I have a theory as to why...
| tuyguntn wrote:
| enlighten us
| logicallee wrote:
| my theory is that their own DeepSeek writes the code for
| them, so they are highly productive.
| menaerus wrote:
| That would be terrifying in itself if true because for
| this type of work you really need the best of the best.
| But I doubt this is the case here. LLMs as we know them
| today are not quite yet there for this type of work.
| genewitch wrote:
| Do you, though? Need the best of the best?
| bee_rider wrote:
| As someone who did some simulation focused engineering
| grad school stuff; there is a tendency for some of the
| best to go become quants. Does the field need it? I don't
| know. But for whatever reason the draw of the "print
| money using math tricks" seems to attract some hardcore
| folks, haha.
|
| It is really frustrating to see good engineers go to play
| trading games. We should study how exactly it is China
| managed to unlock this capacity.
| htrp wrote:
| Government effectively banned unproductive tech (adtech,
| fintech etc) and told ppl to go do stuff like robotics
| and AI
| bee_rider wrote:
| I'm beginning to suspect this invisible hand isn't as
| clever as it was made out to be.
| menaerus wrote:
| I think so yes. There's very few engineers that can pull
| out this type of work IME. In a pool of ~30M SEs around
| the globe I'd say there's no more than ~30K of such
| engineers. This is 0.001% and it's a very optimistic
| number I'd say.
|
| Why do you think this would be controversial? This isn't
| an every day work.
| bee_rider wrote:
| I'd expect them to highlight that, rather than keep it
| secret. I wouldn't be surprised if they used it a little
| bit, but probably not to an extent that is really unique
| or unusual.
| codydkdc wrote:
| lol
| reissbaker wrote:
| They wrote this in 2019, well before any useful codegen
| LLMs existed.
|
| https://www.high-flyer.cn/blog/3fs/
| digdugdirk wrote:
| 996 work culture?
| Forbo wrote:
| That's been illegal for three and a half years?
| rfoo wrote:
| That's been illegal since May 1995 (before that China had
| six working days week).
|
| Does it really matter whether it's illegal or not, if
| there is no enforcement? Pinduoduo (in other name, Temu)
| has been doing 70 hours week since they started. Yes,
| they are still doing it right now.
| vitaflo wrote:
| To be fair they've been working on this since 2019 for HFT. So
| it's not like they just whipped this up.
| bee_rider wrote:
| The amount of brainpower wasted on HFT games that will never
| see the light of day is kind of a bummer. Congrats to China.
| thohj4234234324 wrote:
| This is very humbling.
|
| OpenAI et. al kind of have also been very deep down the systems
| rabbit hole (eg. Triton), but I can't think of anyone else
| (outside of Google/Facebook) who pay this amount to attention to
| things.
|
| Great work; hope Deepseek does even more awesome things going
| forward.
| richardw wrote:
| I've assumed that it's partly because the company has done a
| lot of HFT, which is very focused on performance. But I'm not
| an expert in either.
| WiSaGaN wrote:
| Indeed, the blog mentioned in the other comment showed part
| of 3FS code was completed at least since 2019, when this was
| still a project of the quant funds. In HFT, you tend to
| dogfood a lot of the things to achieve low latency, high
| performance, sometimes just because HFT system just need to
| do one specific thing, and those off the shelf stuff usually
| cater for a lot wider scenarios where HFT doesn't really care
| about. Here you see similar case which they focus
| specifically on loading large amount of data during training,
| and implement that to the extreme.
| jauntywundrkind wrote:
| Man, 6.6TB/s across 180 nodes is 300Gbps/node, or 37.5GBps.
|
| That's with 14 unnamed SSD per node. I wonder how this would
| scale to higher end SSD,dealing from PCIe 4 to PCIe 5 or PCIe
| 6... Particularly whether one could scale down!
| WithinReason wrote:
| Why is this even necessary? Can you just shard your training set
| to the training nodes ahead of time instead?
| wenyuanyu wrote:
| No, besides accessing training data, there is also logging and
| checkpointing... When you run k8s over it, and there are
| multiple training jobs... isolated local storage is a
| nightmare...
| yalogin wrote:
| It's not clear to me where and how the current popular systems
| fall short. Do they talk about I anywhere?
|
| Also, what specifically is the data access patterns for training
| and inference that are different from traditional use cases?
| jpgvm wrote:
| Well current popular systems are pretty much limited to Lustre
| and the new kid Weka, mostly Lustre though tbh.
|
| You can try to use "standard" options like
| MinIO/Ceph(RADOS)/SeaweedFS but you will very quickly learn
| those systems aren't remotely fast enough for these usecases.
|
| AI training is what this is used for, not inference (which has
| absolutely no need for any filesystem at all). What makes the
| workload somewhat special is that it's entirely random read and
| not cacheable at all as most reads are one and done.
|
| Would Lustre be perfectly fine at 6TiB/s? Yes. Is it a huge
| pain in the ass to operate and make remotely highly available?
| Also yes. If this thing is capable of the throughput but easier
| to operate and generally more modern and less baroque it's
| probably an improvement. TLDR is Lustre is fast but that is
| literally it's only redeeming quality. I have lost far too many
| hours of my life to the Lustre gods.
| cyanf wrote:
| They're using the FS for caching the KV caches of past
| requests. It's why they're able to charge so little on prompt
| cache hit.
| jpgvm wrote:
| Ahh I missed that. Yes prefix caching and RAG are 2 cases
| were you will want something like this during inference
| time.
| rfoo wrote:
| > What makes the workload somewhat special is
|
| I'll add that latency also doesn't matter that much. You are
| doing batched data loading for batch n+1 on CPU when GPUs are
| churning batch n-1 and copying batch n from host memory at
| the same time.
|
| So as long as your "load next batch" doesn't run for like >1s
| it would be fine. But one single "load next batch" on one
| worker means thousands (if not more) random read.
| pella wrote:
| related research paper (english - html ) -
| https://arxiv.org/html/2408.14158v2
|
| arXiv:2408.14158v2 [cs.DC] 31 Aug 2024
|
| _" Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-
| Design for Deep Learning"_
|
| Abstract:
|
| _" The rapid progress in Deep Learning (DL) and Large Language
| Models (LLMs) has exponentially increased demands of
| computational power and bandwidth. This, combined with the high
| costs of faster computing chips and interconnects, has
| significantly inflated High Performance Computing (HPC)
| construction costs. To address these challenges, we introduce the
| Fire-Flyer AI-HPC architecture, a synergistic hardware-software
| co-design framework and its best practices. For DL training, we
| deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved
| performance approximating the DGX-A100 while reducing costs by
| half and energy consumption by 40%. We specifically engineered
| HFReduce to accelerate allreduce communication and implemented
| numerous measures to keep our Computation-Storage Integrated
| Network congestion-free. Through our software stack, including
| HaiScale, 3FS, and HAI-Platform, we achieved substantial
| scalability by overlapping computation and communication. Our
| system-oriented experience from DL training provides valuable
| insights to drive future advancements in AI-HPC."_
| codingwagie wrote:
| I think the difference between deepseek and OpenAI/Anthropic is
| one of the difference between practitioners and academics.
| Ofcourse there is world class talent at OpenAI. But there are
| also alot of "I went to Harvard and want to work in AI", and
| those types of people just simply dont have the technical
| exposure to even think of building something like this.
| mustpax wrote:
| Someone should write a blog post about the
| prestige/effectiveness negative feedback loop. This is also the
| Achilles heel of top tier SV VCs including YC.
| dvaun wrote:
| Can you expand on this?
| djtango wrote:
| You mean this one?
| https://news.ycombinator.com/item?id=9125816
| bugglebeetle wrote:
| The problem isn't the prestige it's that prestigious
| institutions in America don't produce high-quality talent.
| They're instead mostly corrupt credentialing mills for the
| rich and well-connected. From what I understand, DeepSeek
| also only hires from the best universities in China, but
| "best" actually means something relative to how difficult
| entrance to those organizations is to achieve and their
| coursework.
| tway223 wrote:
| I would say most if not every large company in China has their
| own AI infra stack, partially because tech talent is relatively
| more abundant and partially some of the tech leads have been
| exposed to western tech via open source and work experience so
| they have a good success rate (which makes it a more common
| practice). Anecdotally, specifically Google, FB ex-employees
| from oversea offices, MSFT and Intel ex-employees from their
| China offices could be the key elements for this trend in the
| past two decades (Google left China around 2010).
|
| The infra work is usually technically tedious so I think it may
| become some lost art in the west just like those manufacturing
| jobs.
| smallmancontrov wrote:
| As opposed to the US, where every large company has its own
| AI infra stack, often extending down to the silicon and up to
| large open source projects?
|
| What's going on here, why are people forgetting what's around
| them? Does familiarity breed contempt? Are attention spans so
| shot that failure to participate in this week's news cycle is
| enough for "out of sight, out of mind"? Or is HN full of
| Chinese bots now?
| robotnikman wrote:
| Makes me wonder where is the best place to learn how to put
| together and operate something like this then? Certainly there
| should be resources out there somewhere to teach yourself?
| cma wrote:
| Weren't the flash attention authors not just from academia but
| in academia at the time?
| whalesalad wrote:
| The throughput on those charts is pretty wild - multiple
| terabytes per second.
| hintymad wrote:
| A distributed file system is honed as one of the trickiest
| software to write, and we are usually advised not to write a file
| system from scratch (even on top of FUSE), let alone a highly
| optimized one. When a silicon value company is having the 100th
| meeting to align god-knows-what, a team of fewer than 60 already
| came up with a production-grade highly efficient parallel file
| system.
|
| Have we in the valley companies lost touch?
| htrp wrote:
| > team of fewer than 10
|
| the highflyer team are pretty well resourced.... think they
| have more than 10 people
| hintymad wrote:
| Thanks! Updated to 60 per their author list in their paper.
| ssivark wrote:
| Why do you assume everyone in the company (including folks
| working on infra) are authors on the paper? That's
| possible, of course, but isn't it unlikely?
| hintymad wrote:
| That's the only information I have. That said, High-Flyer
| had about 160 people, total, in 2021. Given that F3 was
| in production in 2019, 60 people is a generous
| estimation.
| rvz wrote:
| Once again, DeepSeek continues with another home run.
|
| Can't wait to see what they release next. DeepSeek should be
| studied carefully.
___________________________________________________________________
(page generated 2025-02-28 23:01 UTC)