[HN Gopher] Littlefs - a little fail-safe filesystem designed fo...
___________________________________________________________________
Littlefs - a little fail-safe filesystem designed for
microcontrollers
Author : pabs3
Score : 114 points
Date : 2023-12-11 00:15 UTC (22 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jpcfl wrote:
| Very cool. FYI, I had to remove some compiler flags and change
| the 'section' attributes to get the tests building with Clang,
| but it seems to be working on my Mac.
|
| I was looking for a benchmark that would give me an idea of the
| RAM/ROM footprint for the library, but I don't see one.
| aappleby wrote:
| A caveat for anyone wanting to use this - while the library does
| work and the test suite seems thorough, the implementation does
| some _very_ sketchy things like forcibly casting pointers to
| unrelated types so they can be shoved through some multiply-
| recursive functions and then cast back to the correct type later.
|
| A team I was on a while back ended up abandoning LittleFS as we
| couldn't fully trust the C implementation and my two separate
| attempts to port it to Rust both proved futile (_everything_ had
| to be unsafe).
| thenewwazoo wrote:
| What do you recommend instead?
| aappleby wrote:
| We ended up going with a very trivial system based on tar
| files and bitfield allocators and delegated the more
| filesystem-y like functionality to a higher level of the
| stack.
| yjftsjthsd-h wrote:
| I happen to be in the middle of writing a simple tar
| implementation, and had in fact just started playing with
| the idea of just using it as an on-disk filesystem format.
| The caveats I can see are that 1. it's approximately
| append-only[0], and 2. you have to replay the whole thing
| in order to make sure you have the latest version of a file
| (I'm leaning towards doing so on mount because then I can
| also build an in-memory filename-to-block-address cache).
| Have you hit any issues, or is this as surprisingly
| reasonable as it sounds?
|
| [0] I mean, if the new version of a file fits in the same
| number of 512-byte blocks as the old version you _could_
| update in place, but that 's an unreliable condition at
| best and also guarantees data corruption if you lose power
| halfway; really going append-only makes the implementation
| simple and also makes it easy to be really resilient.
| aappleby wrote:
| I'm not on the team anymore so I don't know if they kept
| the tar stuff around, but it was straightforward to scan
| the flash for valid tar header blocks to build the
| allocation bitmap and then flag header blocks as invalid
| when files were deleted. The total number of files we
| needed to keep in the flash was small, so simple brute-
| force stuff worked fine.
| e63f67dd-065b wrote:
| Not so sure about the tar part, but the append-only
| nature is actually really well-known: look up log-
| structured filesystems, where all operations append to
| the log and GC comes to clean up stale data later. I'm in
| the middle of writing a log-structured disk, and it's
| really nice to work with. You get many things like
| clones, snapshots, etc for free.
|
| The replay problem is usually solved with periodic
| checkpoints of your map, and a checkpoint on shutdown.
| This way you only ever replay on failure, at which point
| performance doesn't matter as much.
| yjftsjthsd-h wrote:
| Yeah, knowing there were filesystems designed like that
| on purpose made me feel better about it:) And once you
| get over the initial weirdness, the ability to get any
| version of any file is pretty nice. Though my favorite
| angle is that if you can live with just burning space[0],
| it makes the thing _so_ easy to implement:)
|
| [0] To be fair, a huge caveat.
| mschuster91 wrote:
| > The caveats I can see are that 1. it's approximately
| append-only[0], and 2. you have to replay the whole thing
| in order to make sure you have the latest version of a
| file
|
| Hmm. I'd go and build a separate metadata/index file that
| you keep updated on each write(=tar-append) operation.
| Grow the tar from the beginning of the underlying block
| device, and the metadata from the rear, and have a
| pointer in the filesystem header that points to the
| current metadata address.
|
| That way, everything is atomic, you avoid the RAM penalty
| from having to maintain the metadata mapping, and the
| only operation that can result in damage in a power-loss
| event is when the write of the filesystem header messes
| up - but that can easily be reconstructed by walking both
| the tar stream and the metadata stream from the beginning
| to the last element that has a valid header.
| reycharles wrote:
| I helped build a tar-backed filesystem for MirageOS[0].
| It is definitely easiest to make it append-only. I did
| add in-place renaming, updates (of file content), pre-
| allocating a file and removing the last file.
|
| We also scan the filesystem on boot and keep a mapping
| in-memory of file names to file metadata and block
| offsets, and update this memory representation when
| modifying the on-disk tar filesystem. If you have _a lot_
| of small files this is maybe not a great idea as you will
| be storing most in memory, though.
|
| For the purpose we had in mind it worked fine: A content
| addressable mirror of package archives. The archives
| exist somewhere on the web and the package file has a
| link to the archive and a checksum. We can then download
| the package and store it by the checksum. This gives
| integrity that the tar filesystem does not offer.
| Removing the last file works great if you only have one
| download job. Otherwise if a download fails and it is not
| the latest file you can rename it to
| `failed/<hash>-<random>` and do garbage collection in the
| filesystem at a later point.
|
| [0]: https://github.com/mirage/ocaml-
| tar/blob/main/mirage/tar_mir...
|
| Update: interestingly, this was motivated after trying to
| use a (mostly) LittleFS-compatible filesystem which
| unfortunately didn't work very well for us at the time
| (bugs, poor performance).
| crest wrote:
| I would recommend thinking long and hard if you really need a
| file system on a microcontroller. What do you intend to use
| the file system for? If it's only for read-only assets like
| images or sounds use a linker script to collect them into a
| sorted set. If you want to write logs a ring buffer is
| faster, simpler and more resilient to corruption.
| Configuration settings are small enough and don't too
| frequently in most systems so a ring buffer or just a pair of
| double buffered flash pages with a checksum and version
| counter can be enough.
|
| LittleFS is tries to hit a sweet spot that only exists on
| MCUs with NOR flash and relatively much RAM and fast CPU
| cores. The problem is that it's complex compared to the
| alternatives. The two most common alternatives I can think of
| are FAT which will wear out flash given half a chance and
| SPIFFS which is simpler and slower than LittleFS.
|
| FAT is a terrible file system for writing on a MCU, but
| because of its history as the file system for floppy disks
| it's still supported by every desktop operating system. This
| makes it perfect to allow users to edit configuration files
| stored on a microSD or replace assets like background images
| without having to use specialised tools.
|
| The annoying answer: it depends.
| HocusLocus wrote:
| Great points. I just want to toss out an example of why you
| might want an OS to use with flash of decent size to do
| wear leveling on everything. The classic example I give is
| a media player subject to power interruption and you don't
| want to include an internal battery in the design for
| graceful powerdowns, so the player needs to save a
| checkpoint of play state every ~5 seconds so the device can
| resume intuitively. Such as indexing back to your position
| in a 40 hour audiobook. You're also going to have menu
| navigation and a combination of changing settings and
| device local media library changes. And now comes along
| some Internet angle so there are C libraries that implement
| browser functions and that means cache and cookie writing.
| And software updates. Ring buffers are smart but there's a
| whole lot of writing going on here! It would be great to
| code everything in high levels so if it emulates properly
| on a desktop, once it's pushed into the devices (say a
| generous 128GB flash with 28GB reserved for the system) the
| devices will have a 20 year lifespan with no batteries to
| fail or leak.
| SV_BubbleTime wrote:
| I can tell you take offense to the questionable casting... but
| did you discover anywhere it made mistakes and failed - or you
| just didn't like it?
| dnedic wrote:
| Undefined behavior will bite you eventually. Updated the
| compiler? You have no idea what to expect. Changed lines of
| code related to that part of code? Again, no idea what to
| expect.
| jay-barronville wrote:
| While I agree that undefined behavior should be avoided,
| it's not as rigid as you put it. Just because something is
| undefined behavior by specification (or lack thereof)
| doesn't mean it's undefined behavior in practice. As a
| general rule though, I do agree with you.
| crest wrote:
| Casting pointers to the correct integer types and again
| back isn't UB, but several things you can you can easily do
| along the way are at the very least implementation
| specific.
| hinkley wrote:
| What prevents you from taking the underlying data structures
| and reimplementing the API?
| zoomablemind wrote:
| > ...the implementation does some _very_ sketchy things like
| forcibly casting pointers to unrelated types so they can be
| shoved through some multiply-recursive functions and then cast
| back to the correct type later.
|
| I did not look into the code, however such casts can be a
| reasonable way to implement polymorphism in C.
|
| Of course, having some common type field would be more robust,
| instead of blindly casting whatever pointer comes at input.
|
| For example, such type could be defined as first field in the
| 'base' structure. Thus the cast into, say, `int` should yield a
| valid/expected type value, which would be checked in the called
| functions before casting into the expected type.
| tom_ wrote:
| Pointers correctly round-trip through unrelated pointer types,
| provided the alignment for both types is compatible.
|
| At least one bit does look a bit questionable: lfs_mlist is
| treated as the common initial sequence of lfs_dir and lfs_file,
| even though it isn't, and common initial sequences only apply
| to union fields anyway. Example cast of struct dir * to struct
| lfs_mlist * (probably valid of itself, assuming the alignment
| is compatible): https://github.com/littlefs-
| project/littlefs/blob/c733d9ec57... then use struct dir as if
| it were actually a struct lfs_mlist:
| https://github.com/littlefs-project/littlefs/blob/c733d9ec57...
|
| (There's other occurrences of the same kind of thing.)
|
| Strictly speaking, I think this might be unfixable without a
| bunch of work, but so much stuff does this kind of operation
| that any compiler that doesn't do what you expect will have
| been fixed by now. (Assuming you're not one of those people who
| is going to pop up and tell us with a straight face that what
| we _should_ expect is for the compiler to do absolutely
| _anything_ - except, perhaps, for having it generate correct
| code, which would be defective behaviour that should be
| eliminated.) Maybe improve the odds by using lfs_mlist as the
| common initial sequence of both structs, and fingers crossed
| that the compiler considers the union rules to apply to this
| case too. Or compile with -fno-strict-aliasing.
| hdgr wrote:
| We use littlefs in Flipper Zero's firmware[1] for storage in
| leftover flash space after the main firmware image. Flipper
| implements a virtual FS, where both external SD card and internal
| storage have own mount points. SD card is used for storing apps
| and user-created data, and internal littlefs contains persistent
| data like BLE pairing, system services' configs and such.
|
| LFS has neat features like wear leveling and optimizations for
| storing tiny files directly in their parent directory's data
| structures[2].
|
| We never had any issues with littlefs - however, it cannot be
| easily resized when amount of available leftover space changes
| with firmware updates. So on installing an update, it gets fully
| backed up to SD card, reformatted and later restored.
|
| [1] - https://github.com/flipperdevices/flipperzero-
| firmware/blob/...
|
| [2] - https://github.com/littlefs-
| project/littlefs/blob/master/DES...
| quadhome wrote:
| We used littlefs on an embedded project. Everything worked as
| advertised! Solid.
|
| We kept up with all their releases until their latest API changes
| forced us to fork to keep our anti-glitching mitigations.
___________________________________________________________________
(page generated 2023-12-11 23:02 UTC)