[HN Gopher] The thinking behind the 32GB Windows Format limit on...
___________________________________________________________________
The thinking behind the 32GB Windows Format limit on FAT32
Author : samizdis
Score : 77 points
Date : 2021-01-04 12:14 UTC (10 hours ago)
(HTM) web link (www.theregister.com)
(TXT) w3m dump (www.theregister.com)
| wazoox wrote:
| Actually NT4.0 did not support FAT32 at all (unless you installed
| a little-known driver from Sysinternals I've never actually seen
| used in the wild).
|
| So maybe his work was on Windows 2000?
| BuildTheRobots wrote:
| I hit this a couple of days ago when helping a friend set up
| their new DJ decks. The decks only supported FAT32 but would
| happily read a 128gb USB stick.
|
| The article states that you can format as fat32 from the command
| line - this wasn't my experience. Disk Util (cli) saw my raw
| partition and started formatting it as FAT32, but seemed to fail
| about an hour in saying the disk was too big.
|
| I ended up finding a random abandoned GUI utility which seemed to
| work, though it's extremely infuriating.
| woliveirajr wrote:
| TL;DR: "simple lack of foresight combined with the age-old
| problem of the temporary solution becoming de-facto permanent"
| simonblack wrote:
| Exactly. Just another case of "640K should be enough for
| anybody".
|
| But we have to admit that storage has leapt hugely in size _and
| very rapidly_. In 1990 or thereabouts, a common hard disk size
| was about 30 megabytes. Today I have _files_ that are many
| times that size, and we think that that is quite normal. A mere
| five or so years later around 1996, that size had jumped to
| 4000 megabytes! These days my laptop has 6 million megabytes.
|
| We can't blame filesystem planners for lack of foresight. They
| were blind-sided by progress just like the rest of us were.
| ben509 wrote:
| He had already seen exponential growth in storage capacity
| and could easily extrapolate from that. Moore's law was
| coined in the '60s, after all.
|
| This wasn't a lack of foresight, this was being prudent. If I
| can't come close to testing an upper limit, it's correct to
| acknowledge it may degrade in a way I haven't anticipated.
|
| Especially since he was writing a GUI which should generally
| avoid handing users footguns, and which can be updated over
| time.
| mnw21cam wrote:
| I have a single file that is 6 million megabytes in size.
| Thristle wrote:
| who/what reads that file?
| petschge wrote:
| Not my file, but I have produced HDF5 files in that size
| range using simulation codes on supercomputers and
| paraview can open them and render 3d view as video from
| it (eventually).
| mnw21cam wrote:
| tar
| AnIdiotOnTheNet wrote:
| That stretches the definition of "a single file" when it
| is basically a filesystem itself.
| bilekas wrote:
| Not necessarily, it's all relative really. But the point
| is, the limitations or expected limitations are only
| relevant until they're not..
|
| Also why not tar a drive for archive if I want :}
| jcranmer wrote:
| I haven't managed a 6TB file before, but I did have a
| >100GB log file I ended up parsing with awk just to be
| sure that the memory consumption on the consuming process
| wouldn't go crazy.
| jandrese wrote:
| More directly: He only expected his code to be used in Windows
| NT and made the fatal flaw of thinking that just because he
| slapped it together in an hour it would be easy for someone to
| rewrite in the next version of Windows to up the limit.
|
| Instead the code sat untouched for decades as the limit slowly
| became more and more unreasonable.
|
| IMHO the real issue is the artificial limit. Instead of
| straight up disallowing you from creating large filesystems, it
| should have had a dialog box warning about small files being
| inefficient but letting you make it anyway.
| bluedays wrote:
| The original video this article is referencing is here if anyone
| wants to just watch it instead:
| https://www.youtube.com/watch?v=bikbJPI-7Kg
| ike77 wrote:
| Maybe it's not for the worst, as otherwise there might have been
| less push for exFAT and this inefficiency would have persisted
| much later..?
| selimnairb wrote:
| Hopefully exFAT implementations will get more robust. At least
| with the macOS implementation, it has always been prone to
| reporting corruption when devices are not ejected properly. I can
| usually fix it by running check disk or whatever on Windows, but
| sometimes not.
| MrDOS wrote:
| I've had more filesystem corruption trouble with exFAT on Linux
| than with any other filesystem, ever, including Btrfs. Really
| crossing my fingers and hoping for the quality of the Samsung
| driver in Linux 5.7. It's going to be great to have an option
| for high-capacity, cross-platform read/write storage. The day I
| never have to use NTFS-3G again cannot come too soon.
| kasabali wrote:
| Were you using the FUSE driver?
| MrDOS wrote:
| That's a great question. I kind of skirted around
| indicating one way or the other in my parent post because
| I've forgotten. My troubles were as recent as the beginning
| of 2020, but I encountered them on a Raspbian machine, so
| Lord knows what kernel version it was running at the time.
| Probably pre-5.4, in which case, yes, it would've been the
| FUSE driver.
| kasabali wrote:
| Thanks it was probably the FUSE driver. I haven't had a
| problem with it but I'm not a heavy user.
|
| On the other hand I've been a heavy user of exFat
| formatted sd cards on Samsung phones for long years and
| had no problems with it, so hopefully the new Linux
| driver (which Samsung upstreamed) will be solid.
| brnt wrote:
| So, since we're on kernel 5.10, has this new Samsung exFAT
| driver been tested? Is it at least as reliable as btrfs?
| pantalaimon wrote:
| > The day I never have to use NTFS-3G again cannot come too
| soon.
|
| That day might be soon
|
| https://lkml.org/lkml/2020/12/31/126
| MrDOS wrote:
| I know everyone talks about 2020 having felt like a long
| year, but I had totally forgotten that Paragon had started
| trying to submit that back in August[0]. I'm really glad to
| see this is progressing. What a monumental set of changes.
| This won't change the NTFS situation on macOS, of course
| but even in a post-exFAT world, NTFS will remain a relevant
| concern for Linux/Windows dual-booting, and I'd love to see
| this get merged.
|
| [0]: https://lore.kernel.org/linux-
| fsdevel/2911ac5cd20b46e397be50...
| chungus_khan wrote:
| One change from previous FAT filesystems to exFAT is the
| removal of the second, redundant file allocation table. Disks
| are less unreliable now so it is less necessary, but there are
| still some scenarios, especially when the filesystem is in a
| dirty state, where the table can be corrupted.
|
| This can also really hurt in unexpected ways with otherwise
| badly written drivers. Nintendo's driver for the switch would
| constantly write to the table unnecessarily, and while this
| doesn't do all that much in normal operation (other than waste
| a small number of writes), it made it drastically more likely
| (in some cases for some games almost guaranteed) that the table
| would be partially written and corrupt if the system crashed or
| lost power unexpectedly. When some new Pokemon games came out
| and were initially a bit unstable this came up a lot.
| dbt00 wrote:
| The choice he made was probably a reasonable balance -- the
| alternative was reinventing the filesystem, which was probably a
| bit out of spec for implementing the GUI format dialog. And
| eventually happened, but it took a while.
| Thristle wrote:
| Its not reinventing. Its just setting a limit on max cluster
| slack. Like he says in the video, you can totally use the CLI
| and format any disk size you want, its just a soft block
| doser wrote:
| Many embedded devices can read FAT32 and that's it. If you need
| to use a drive larger than 32GB for these, there are tools that
| will let you format them as FAT32 from within Windows.
|
| One example: FAT32Format GUI -
| https://www.softpedia.com/get/System/Hard-Disk-Utils/FAT32fo...
| kasabali wrote:
| It also conveniently allowed Microsoft to collect license fees
| from ExFAT patents on all SDXC devices.
| creeble wrote:
| What is the practical difference between exFAT and FAT32 when you
| can format FAT32 up to many GB?
|
| Is it just that they are different formats, and exFAT has taken
| over, in terms of pre-formatted USB media?
| BlueTemplar wrote:
| FAT32 also has a 4 Go file size limit (think large video/VM
| files, DVD/Blu-ray images...)
| greghendershott wrote:
| I rarely read The Register. Is that the usual tone -- as if the
| author wishes they were Lady Whistledown, the gossip writer on
| Bridgerton?
| QuercusMax wrote:
| Maybe with a little _Variety_ mixed in, but you 're not far off
| the mark. Definitely in the standard British vein of snarky
| tabloid writing.
| tym0 wrote:
| It is, it's similar to Private Eye in the UK or Le Canard
| enchaine in France.
| arp242 wrote:
| Yes, pretty much.
| m000 wrote:
| TIL: In 2019, Microsoft released the exFAT [1] specs to a large
| group of vendors [2], including prominent Linux vendors. I.e.
| exFAT should now be preferred over FAT/NTFS for media used across
| different operating systems.
|
| [1] https://en.wikipedia.org/wiki/ExFAT
|
| [2] https://en.wikipedia.org/wiki/Open_Invention_Network
| grenoire wrote:
| Huh, I thought exFAT was much older, not 2019.
| MrDOS wrote:
| It is - it dates back to 2006. But before 2019, the spec was
| proprietary and patent-encumbered. And the Linux kernel
| didn't include a good driver for it until 5.7, released in
| May of last year[0].
|
| [0]:
| https://kernelnewbies.org/Linux_5.7#New_exFAT_file_system
| kasabali wrote:
| It is still patent encumbered. MS just promised to not give
| trouble for it being in the Linux kernel.
| MrDOS wrote:
| Right - that's what I meant, but I didn't communicate it
| clearly. Thanks for clarifying.
| gruez wrote:
| >TIL: In 2019, Microsoft released the exFAT [1] specs [...]
| I.e. exFAT should now be preferred over FAT/NTFS for media used
| across different operating systems.
|
| You seem to be very optimistic about how fast these get
| implemented into embedded devices.
| bilekas wrote:
| I would imagine if embedded devices are being assembled and
| configured with filesize concerns, then exFAT would be the
| natural implementation they would opt for. Or do you mean
| updating/transitioning legacy embedded?
|
| The spec doesn't mention backwards compat, but does mention
| as a goal
|
| >Retain the simplicity of FAT-based file systems.
|
| I would like to test out if an input embed system is exFAT
| compat. And what might be required to update.
| numpad0 wrote:
| I believe the implication is SD card/USB drive support on
| microcontrollers without an OS.
|
| SD under 32GB uses FAT32 so you can't drop supports, and
| FAT32 stack is often already pushing the maximum ROM size,
| so no room is left for a novel filesystem. Lots of
| developers aren't themselves proficient at writing
| filesystem code as well, and there aren't as many copy-
| pastable exFAT implementations as there are for FAT32.
|
| So unless they know what they're doing and unless it's
| critical to their product, the majority won't easily move
| onto exFAT.
|
| As for the use within the device "firmware", the user
| defined program is usually just a giant baremetal a.out in
| the middle of the ROM at a specified address, so no FS is
| involved there.
| wongarsu wrote:
| Embedded devices often have a lifespan measured in decades.
| And for short-lived devices like consumer all-in-one
| printers (~4 years) it's not like every new model has a new
| firmware written from scratch. They get whatever the last
| model had, with a few updates for new device features. That
| might include exFAT, or it might not.
| netsharc wrote:
| I have a 2011 car which has a USB port where I can plug a USB
| stick with MP3s (remember those, kids?) on them, I doubt it
| will ever be made ExFAT-compatible.
|
| OTOH, I do have the Windows logo on my steering wheel, and
| somewhere inside the car there's a Windows CE system
| installed (it doesn't have a multi-color LCD screen and
| interacts using speech recognition, so it's not really
| obvious).
| riedel wrote:
| Maybe there are even capable embedded implementations even
| before Microsoft implements all supported features like ACLs
| in their OS.
| Abishek_Muthian wrote:
| Much before MS did that, macOS had builtin exFAT support
| presumably licensed from MS, Paragon had exFAT support for
| android for a one time fee which works with other file manager
| apps(also licensed from MS), On Linux Exfat-utils enabled
| support for exFAT(open-source implementation).
|
| Has there been any changes to these since MS has opened the
| license to exFAT?
| adrian_b wrote:
| The change is that now there is an exFAT driver included in
| the Linux kernel.
|
| Therefore, if you have a recent Linux kernel, you no longer
| need to install a user-mode driver for exFAT and the kernel
| driver has better performance.
| m000 wrote:
| Third-party exFAT support did exist before 2019, but vendors
| had to pay a licence fee to Microsoft. Apple apparently paid
| the fee.
|
| Regarding Linux, exfat-utils only provided limited
| functionality. The kernel mainline lacked a native exFAT
| driver. Mounting exFAT volumes was possible only through FUSE
| (with all its shortcomings).
| kop316 wrote:
| Unfortunately, this only applies to systems that actually have
| this (i.e. they were updated after exFAT released or released
| after and included this update). There will be sadly many
| systems that will never receive this update.
|
| But for a fun fact, if you see many embedded systems that say
| they can only support an SD card up to 32 GB, it is a soft
| limitation due to the fact that media over 32 GB are pre-
| formatted to exFAT (and the system lacks support for exFAT).
| FAT32 can go up to 2 TB. So I have found that reformatting a
| larger media back to FAT32 makes it recognizable to that system
| with all of the storage space.
| est31 wrote:
| > But for a fun fact, if you see many embedded systems that
| say they can only support an SD card up to 32 GB, it is a
| soft limitation due to the fact that media over 32 GB are
| pre-formatted to exFAT (and the system lacks support for
| exFAT).
|
| There might also be the issue of the card reading hardware
| only supprting SDHC instead of SDXC (or SDUC).
| kasabali wrote:
| Even then it is actually only a driver or firmware
| limitation most of the time.
|
| My old laptop has Ricoh SDHC card reader. Windows can't
| read 64GB SD cards (properly formatted with exFAT according
| to the standard) on it because it's using the proprietary
| Ricoh driver.
|
| On Linux it works just fine because it's using the generic
| SD card reader driver which supports SDXC.
| kop316 wrote:
| That is true, though I have yet to run into that issue
| though for the devices I have tried it on (which admittedly
| is not a large sample size).
| martinald wrote:
| FAT32 only supports 4GB files though, so not sure how useful
| this is for a lot of media usecases.
| moftz wrote:
| At one point, that was more than enough to store DVD rips.
| fuzzfactor wrote:
| The patents on FAT32 are now fully expired, including the final
| lingering LongFileName one.
|
| Therefore FAT32 will remain preferred until the patents expire
| on exFAT, even though the specification for exFAT has been
| released, things like that have proven to be not good enough
| until after the patents expire.
|
| The purpose of exFAT appears mainly to be an attempt to foist a
| new encumbered filesystem on developers now that the first
| mainstream Microsoft filesystem is in the public domain.
|
| We will know when Microsoft has reduced their anti-user stance
| on filesystems after they have released very detailed full NTFS
| specifications in the most useful way possible.
|
| Also I have the greatest respect for Dave, and like he
| mentioned he was only doing the UI not the bitmoving of the
| actual Format process.
|
| Those who were developing the NT version of FAT32 formatting
| failed in much more serious ways where Dave did not.
|
| The inconsistency compared to FAT32 formatted by W9x is
| actually intolerable.
|
| Best reliability still requires booting to the DOS from W98SE
| to properly format FAT32.
| adrian_b wrote:
| The main use for FAT32 and exFAT is to transfer files between
| different operating systems or between a computer and some
| dedicated device, e.g. a photographic camera.
|
| They are not useful for anything else except memory cards,
| USB sticks and external SSDs.
|
| Unfortunately, nowadays it is frequently necessary to copy
| files larger than 4 GB. When using FAT32, you cannot do that
| without splitting the file and concatenating the parts at
| destination. That wastes time and it is annoying and
| sometimes it may be even impossible, when the destination
| lacks a concatenation program or an archiver that can deal
| with split archives.
|
| After Microsoft released it, now exFAT is supported in the
| Linux kernel. Because exFAT is also supported by all
| commercial operating systems, it is now the best way for file
| transfer.
|
| It does not matter any more if there are still valid patents
| for exFAT.
| KronisLV wrote:
| > It does not matter any more if there are still valid
| patents for exFAT.
|
| Would you care to elaborate for why that is? Would anyone
| else with knowledge about the patent system and any
| possible risks imposed by it also like to do so?
| m000 wrote:
| > Would you care to elaborate for why that is? Would you
| care to elaborate for why that is?
|
| To put it bluntly, if Microsoft's guarantees are good
| enough for Linus so as to merge the driver in the Linux
| mainline, they're good enough for me too.
| igneo676 wrote:
| Because, effectively, the exFAT filesystem is OSS since
| August 2019 and Microsoft released all the specifications
| to it [0]. Microsoft has essentially taken public, legal
| steps to ensure that anyone else can use exFAT without
| legal encumbrances and without threat of lawsuit.
|
| IANAL - so there's likely some caveats (you probably
| can't just use the exFAT name unless certain
| qualifications are met, just like you can't use Firefox's
| trademarks willy-nilly), but that should be good enough
| for just about anyone. Including paranoid OSS peeps like
| me :)
|
| 0: https://cloudblogs.microsoft.com/opensource/2019/08/28
| /exfat...
| Delk wrote:
| I'm also not a lawyer, but AFAIK publishing a
| specification has nothing to do with whether the tech can
| be legally used without a patent license.
|
| I don't know how the release to Open Invention Network
| affects things, though.
| tmd83 wrote:
| I guess that's the best option for compatibility. But it's not
| journaled and single fat, very prone to corruption right and
| fixing errors could take a while for lager disk like external
| hdd. There's not really any good option in terms of reliability
| and compatibility I guess.
|
| I have had issues with ntfs getting corrupted and losing file
| both in mac and linux at times so there's that risk with
| implementation too.
| rlv-dan wrote:
| He made a decision and assumed when making future NT versions
| this would be revised and updated with the times. I can only
| image that future teams looked at this and thought "someone chose
| 32GB for a reason, better not change it"...
| Zardoz84 wrote:
| Well, Windows 95/98/ME, 2000 and XP shared the same old font
| installer that Windows 3.11 . I wonder why no body changed it
| before.
| macjohnmcc wrote:
| Likely if it ain't broke don't fix it.
| Delk wrote:
| Maybe, but it could also be about just not getting around to
| doing it.
|
| Changes, especially in a large system with a large user base,
| require extensive testing, and even small changes may come with
| a lot of red tape. I don't know how things work at Microsoft,
| but I wouldn't be surprised if making changes to the UI for a
| technology that's being superseded anyway just didn't make
| anybody's list.
| hocuspocus wrote:
| If my memory serves well, when 40+ GB HDDs started being common,
| you were either using Windows 2000 and NTFS already, or it was
| probably a good practice to have several partitions, given the
| overall unreliability of Windows 9x and the not so infrequent
| clean reinstallations. And then came Windows XP and the issue was
| moot.
|
| It's a bit of a shame Microsoft didn't push and open up exFAT
| earlier and allowed FAT32 to become an issue again, only with
| removable flash drives and memory cards this time.
| kasabali wrote:
| > It's a bit of a shame Microsoft didn't push and open up exFAT
| earlier and allowed FAT32 to become an issue again, only with
| removable flash drives and memory cards this time
|
| Erm, because that was the idea? SDXC is a big racket and
| Microsoft has been collecting patent fees from _any_ kind of
| consumer device that 's capable of reading memory cards thanks
| to the filesystem requirements in the standard.
| AnIdiotOnTheNet wrote:
| The problem with removable drives is that no one came up with a
| better filesystem than FAT32 that could be implemented easily
| by everyone. All the *NIX filesystems embedded permissions that
| were useless on non-POSIX systems and caused issues on POSIX
| ones when you actually moved a drive from one computer to
| another. So MS extended FAT again and kept it under its IP hat,
| as was their wont[0] at the time.
|
| [0] Non-native english speakers may be confused by this:
| https://www.merriam-webster.com/dictionary/as%20is%20someone...
| josefx wrote:
| FAT embedded a long/short name mapping algorithm that a
| modern system shouldn't need, but Microsoft had patents on.
| The main feature of FAT is simple native support on all
| platforms, Microsoft never had a financial incentive to add
| support for alternatives on Windows.
___________________________________________________________________
(page generated 2021-01-04 23:02 UTC)