[HN Gopher] Vramfs - GPU VRAM based file system for Linux
___________________________________________________________________
Vramfs - GPU VRAM based file system for Linux
Author : pabs3
Score : 212 points
Date : 2023-02-19 06:43 UTC (16 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| davidy123 wrote:
| Wait'll you guys hear about buying more RAM and using it as a RAM
| disk.
| doublerabbit wrote:
| Woah. Next you'll tell me I could increase the dimension size
| of my punchcard, to store more upon.
| rbanffy wrote:
| You can store twice as many puns on a 16" punchcard.
| anon___ wrote:
| I have more RAM than I need, and just the other day I set up a
| RAM disk to put my /tmp directory on. My comic reader unpacks
| _huge_ rar and zip archives into /tmp at every run, and I
| don't want it to wear out my SSD. I put this line in
| /etc/fstab: none /tmp tmpfs defaults,size=4G
| 0 0
|
| It works like a charm. And in the worst case, if I run out of
| RAM anyway, it readily swaps out.
| bravetraveler wrote:
| This is a commonly recommended thing to do! It helps IPC
| performance in some ways - also makes sure nothing in there
| persists longer than intended.
|
| You might consider these common mount options for a little
| extra security: noexec,nosuid,nodev
|
| _(and potential headaches, to be fair)_
|
| While _/ tmp_ is a great world-writable place, these restrict
| it from being home to executables/devices -- common sources
| of abuse
| anon___ wrote:
| Thank you for the tips! It was just a quick hack and I
| never knew much about the mount options, but I guess it's
| time to learn.
| bravetraveler wrote:
| Certainly!
|
| I picked these up through some compliance benchmarks,
| commonly applied to _/ tmp_ -- I'd exercise caution with
| these elsewhere, they're fairly restrictive
| fwlr wrote:
| I have lived to see man-made horrors beyond my comprehension
| rbanffy wrote:
| That's not too bad. Most desktop work require only a tiny
| fraction of the memory a discrete GPU has on-board. As long as
| the OS manages it well enough (letting programs use the
| remaining memory safely while keeping the VRAM filesystem
| safe), seems like a good solution.
| deathclassic wrote:
| hmmm. Wonder what would happen if you configured a zfs pool with
| this as the l2arc vdev.
| casual-dev wrote:
| Question is, can I use my spare APU for this. On second thought,
| giving it more VRAM in BIOS only means using more RAM for it. So
| it would just be a RAM disk with extra steps.
|
| Nevertheless, neat!
| dusted wrote:
| Am I reading that wrong, or is it slower than 25 Gbps ethernet ?
| redox99 wrote:
| It's ancient hardware, and FUSE is very slow. A benchmark on
| modern hardware would be interesting.
| ptman wrote:
| Throughput likely, but how about latency?
| dev_snd wrote:
| The readme states that currently it transfers around 2
| gigabytes per second which is 16 gigabits per second.
|
| The rest I leave as an exercise to the reader.
| rwmj wrote:
| It would seem more natural to map VRAM to a block device, using
| nbdkit. Then you could create a regular filesystem like ext4 on
| top. Is there a specific reason why writing an esoteric FUSE
| filesystem is better?
| Taek wrote:
| One thing I'm not seeing mentioned is the security implications
| of running a filesystem in VRAM. Unless things have changed, GPU
| VRAM is basically free-for-all and any application can do
| read/write operations anywhere in the memory bank.
|
| There are certain scenarios where this is okay but lots of
| scenarios where it is not (example: webgl)
| tenebrisalietum wrote:
| Use LUKS to encrypt it.
| londons_explore wrote:
| Webgl gets to read memory that isn't its own? That can't be
| right... There would be massive info leaks...
| flohofwoe wrote:
| Yeah, it isn't right. In the very early WebGL days there have
| been a handful of implementation bugs where previous VRAM
| content was accidentally exposed in textures, but these
| things have been fixed quickly. AFAIK, most of this is now
| also taken care of down on the hardware and driver level.
| monocasa wrote:
| Things have in fact changed. GPUs overwhelmingly have their own
| MMU to seperate different user contexts, including as accessed
| through their DMA engines. The few cases under Linux that don't
| such as the Raspberry PI's GPU instead use the kernel to wrap
| any submissions to the GPU where they can abstract and validate
| access to memory.
| nextaccountic wrote:
| How can I search for whether my GPU supports this? My GPU is
| a Radeon HD 8670M, on a very recent laptop
|
| Is this the IOMMU (which isn't generally supported, except on
| high end hardware [0]) or something else?
|
| I think my GPU doesn't support IOMMU
|
| [0] https://en.wikipedia.org/wiki/List_of_IOMMU-
| supporting_hardw...
| monocasa wrote:
| Yeah, it supports it. Pretty much every desktop/laptop GPU
| from the past 15-20 years supports this in fact. I actually
| remember John Carmack lamenting back in ~2009 that these
| MMUs were ubiquitous already and that he wanted to more
| directly use the MMUs to implement virtual memory for
| IdTech4's MegaTexture feature, but couldn't get the GPU
| vendors to expose them in usable ways for application
| developers.
|
| Mantle/Vulkan/DX12/Metal was in a lot of ways a reaction to
| the thought 'well because of the ubiquitous MMUs in GPUs,
| user code can only crash their on GPU context, so it can't
| hurt to give them access to a lower level, video game
| console-esque API. At most they crash themselves and that
| was always allowed'.
|
| > Is this the IOMMU (which isn't generally supported,
| except on high end hardware [0]) or something else?
|
| No it's another MMU on the GPU itself. An IOMMU when
| combined with a GPU is a third MMU between the GPU and
| physical memory so you can give more or less raw GPU access
| to a VM securely.
| nextaccountic wrote:
| ... but why does, when my X11 (happens in wayland too
| iirc) system has some kind of bug, one window displays
| the contents of another (with some graphical corruption)?
|
| my supposition was that a buffer of one program ending up
| in a VRAM associated with another program can only happen
| if there is no MMU, otherwise, what's the point
|
| also: do you happen to know why do Qubes folks think that
| it's okay for two mutually untrusted programs run on the
| CPU, but can't accept that two untrusted programs can run
| on the GPU without eveasdropping on each other? even with
| IOMMU it seems.
|
| it gives me the impression that safely sharing the GPU is
| a work in progress / not very secure
| userbinator wrote:
| VRAM is accessible via an aperture in physical address space (I
| believe it's bank-switched for GPUs with more VRAM than the
| aperture size), which normally isn't directly accessible to
| user-mode applications, much like most hardware on Linux.
| flohofwoe wrote:
| WebGL goes to great lengths to not expose any 'junk' VRAM data
| or allow out-of-bounds access even on old GPUs. But for the
| last decade or so, GPUs and their drivers implement the same
| memory protection mechanisms as CPUs (and AFAIK WebGL was a
| major driving force in that area).
| riobard wrote:
| would be interesting to combine this with `zram` to move
| compressed swap to vram instead
| drpixie wrote:
| and you have all that GPU processing power for compressing &
| decompressing :)
| rbanffy wrote:
| But then you'd need to transfer uncompressed data over the
| PCIe bus.
| codeflo wrote:
| I assume you're tongue in cheek, but just to make a serious
| point, it should never make sense to swap from cheap RAM to
| more expensive RAM. The whole point of swap is to move rarely
| used stuff into storage with a lower cost per GB. Otherwise,
| you would just buy more RAM in the first place.
| riobard wrote:
| You need to factor in the compression ratio to see the
| benefits of swapping from RAM to RAM :P
| ZiiS wrote:
| If you pay for GPU memory for the 10% of tasks you need it;
| it is cheaper then system RAM the rest of the time.
| userbinator wrote:
| VRAM will still be faster than HDD.
| unphased wrote:
| It's relatively fast (probably could be faster than NVMe if
| optimized, i did not rtfm) and it is available. it's not the
| most horrible of ideas. I am fairly fairly certain lots of
| applications will load assets into vram and then evict them
| from system memory. Why not leverage this for general purpose
| use if it makes sense (after taking into account the security
| implications of course)?
| dspillett wrote:
| _> you would just buy more RAM in the first place_
|
| Unless you have hit the limit on how much "normal" RAM the
| current machine(s) can have/address, and/or you need extra
| memory for something _now_ , etc.
|
| Or, of course, you are just playing with something cool but
| ultimately pointless!
| alexvoda wrote:
| For example, you might have a laptop with soldered RAM and
| a discrete GPU. There may be tasks where you don't need the
| dGPU but could use the extra RAM.
| citrin_ru wrote:
| Unless you have a notebook/desktop with a powerful GPU and
| there were no option with all things staying the same but a
| cheaper GPU so you ended up which under-utilised GPU.
| KRAKRISMOTT wrote:
| Intel Optane anyone?
| peter_d_sherman wrote:
| Agreed!
|
| It would be interesting to implement this and a highly
| parallelizable GPU background data compression algorithm...
|
| It could be zram (zram would be a good choice!), it could be
| something else, it all depends on what kind of
| performance/compression is desired, and how parallelizable the
| compression algorithm is...
| userbinator wrote:
| The idea of using VRAM as general-purpose storage has been around
| for over a decade, although it usually makes more sense to put
| swap on it:
|
| https://forums.gentoo.org/viewtopic-t-395234-start-0.html
|
| https://www.pmenier.net/dotclear/docext/mtd/TIP_Use_memory_o...
| actionfromafar wrote:
| I used I think 32 megabytes VRAM as swap _way_ back when. I
| think it was on single channel AGP or PCI bus, so it wasn 't
| fast, but much faster than disk, so pretty good.
| mobilio wrote:
| Or this one
|
| https://wiki.archlinux.org/title/Swap_on_video_RAM
| dahart wrote:
| I don't know about Linux these days, but Windows WDDM likes to
| have host-backed storage for VRAM allocations... which makes me
| wonder if using VRAM for swap could ever get you into a weird
| recursive state with allocations or swapping. It's already the
| case that allocating VRAM on Windows can require waiting on
| swapping some VRAM to disk. What if that swap had to then go
| back to VRAM?
| Rediscover wrote:
| OMG! I run in either text mode or very minimal graphics mode.
| This is a very appreciated comment. Now I am going to
| reconfigure my home machines to steal back that otherwise
| unused VRAM for swap.
|
| Thank You!!
| chronogram wrote:
| Why would you add a dedicated graphics card to computers that
| run in text mode or a very minimal graphics mode? Seems
| expensive.
| Rediscover wrote:
| I did not add a card, that is the way the machine is
| shipped/manufactured.
|
| SBC.
| 2III7 wrote:
| For non-graphical computing tasks that require a GPU.
| LoveMortuus wrote:
| +extra ports for monitors
| slug wrote:
| Is this safe ? Had the impression that undetected bit-flips
| were more common with GPU RAM than with CPU RAM banks, unless
| you have professional GPU cards with ECC.
| speed_spread wrote:
| I guess you could have software error correction and still
| be faster than swap to flash?
| FooBarWidget wrote:
| In the era of huge ML models, wouldn't it make more sense to do
| the opposite, i.e. use main RAM or SSD as swap voor VRAM? So
| that we can run e.g. ChatGPT on consumer hardware.
| rbanffy wrote:
| A lot of low-end GPUs do that and, when you don't have enough
| memory to fit the working dataset in the GPU's very fast RAM,
| the CPU will end up swapping in and out of the GPU via the
| PCIe bus.
| brucethemoose2 wrote:
| This is easier said than done, though certain frameworks
| (like Microsoft's Deepspeed) try to do this "automatically"
| jwilk wrote:
| Over _two_ decades, as you pointed out in
| https://news.ycombinator.com/item?id=11559869.
| userbinator wrote:
| ...and that was almost 7 years ago too, during which my
| second link there also died; but that was before the rise of
| all the AI stuff and its accompanying demand for GPUs with
| otherwise insane amounts of VRAM.
| gruturo wrote:
| Make that three decades ago. In 1992 I was using a utility
| called VGADISK to create a 176K ramdisk in the VGA memory -
| this was on a 286 "laptop" (rather a luggable) with 1MB of
| RAM, and allowed me to fit entire programs in there as long
| as they didn't use VGA graphics (which would instantly
| corrupt the content of the ramdrive).
|
| Still online thanks to
| http://www.retroarchive.org/garbo/pc/memutil/index.html - a
| whopping 8KB zip archive, so about 30 seconds on my 2400bps
| modem (assuming V42 or MNP4)
| int0x80 wrote:
| Neat! The implementation is simple, clean and well structured --
| a good way to understand in more detail how a FUSE filesystem is
| implemented.
| RobotToaster wrote:
| Just need to use it with a thunderbolt to pcie adapter and a
| graphics card, put swap on it, and you can finally have external
| RAM.
| vatys wrote:
| One step away from downloading more RAM
| redox99 wrote:
| You can always mount an FTP server and put your swapfile
| there.
| jwilk wrote:
| https://downloadmoreram.com/
| ZiiS wrote:
| Prior art http://www.primrosebank.net/images/ZX81M64k.jpg
| codeflo wrote:
| I don't know if it's esoteric filesystem week on HN, but I love
| it.
|
| [1] Yesterday: TabFS - a browser extension that mounts the
| browser tabs as a filesystem,
| https://news.ycombinator.com/item?id=34847611
|
| [2] The day before: Linux's SystemV Filesystem Support Being
| Orphaned, https://news.ycombinator.com/item?id=34818040
| wpietri wrote:
| Holy moly! TabFS is amazing and I missed it. Closing a tab by
| _rm_ ing the dir is especially great. Thanks for mentioning it!
| OscarCunningham wrote:
| It's older than this week, but you might also enjoy 'Harder
| Drive': https://youtu.be/JcJSW7Rprio.
| loeg wrote:
| Tom7 is a national treasure.
| Svenstaro wrote:
| I once posted memefs:
| https://news.ycombinator.com/item?id=17799547
|
| I didn't get picked up very much, though.
| pyinstallwoes wrote:
| That's really cool
___________________________________________________________________
(page generated 2023-02-19 23:02 UTC)