ZFS on RaspberryPi for home backups
reddit / BSD

----------------------------------------------------------------------
crest_ · 2016-02-21 17:36

Running ZFS without ECC isn't worse than running and previous
generation file system under load without ECC, but running ZFS in a 32
Bit address space is just asking for real world trouble. Even if the
user restricts the ZFS caches as far as possible I expect the system
to crash after a few days under load because of heap fragmentation. At
least that was my experience with ZFS on i386.

----------------------------------------------------------------------
[deleted] · 2016-02-22 03:52

[deleted]

----------------------------------------------------------------------
phessler · 2016-02-22 13:21

do the Pi's have enough ram for zfs?

----------------------------------------------------------------------
daemonpenguin · 2016-02-22 14:07

Yep, I've been running a Pi with ZFS for about a year. Works like a
charm. ZFS really doesn't require nearly as much RAM as people often
claim.

----------------------------------------------------------------------
daemonpenguin · 2016-02-22 14:08

I've used ZFS with 32-bit machines/OSes for about seven years without
seeing any instability. It might be another factor that is causing
crashes on your systems.

----------------------------------------------------------------------
anotherhue · 2016-02-22 15:59

Without deduplication they do.

----------------------------------------------------------------------
crest_ · 2016-02-22 19:30

Did you ever try to run `make -sj4 -C /usr/src buildworld` and `rsync`
at the same time? The system had enough swap but panic()ed anyway
because there was no continuous chunk of free memory large enough to
hold a 128KiB ZFS block.

----------------------------------------------------------------------
daemonpenguin · 2016-02-22 20:08

That sounds like a low memory issue then, not a 32-bit issue. If your
machine doesn't have enough RAM to do both then it won't matter if you
are using a 32- or 64-bit OS.

----------------------------------------------------------------------
crest_ · 2016-02-22 20:13

You can limit the ZFS memory consumption at the cost of increased I/O
overhead incurred by the COW design. I ran ZFS with 768M on a VIA C7
for a while out of curiosity. After a lot of careful tuning the system
still wasn't rock stable e.g. it still didn't like rsync while under
memory pressure. The main problem was the *CPU* overhead. You won't
notice it with modern CPUs but ZFS has a lot more CPU overhead
compared to UFS. It checksums all the data (including metadata) and
the code is heavy on 64 Bit (and 128 Bit) integer operations. Running
this code through a scalar 32 Bit core with small caches and single
DDR2 memory channel really tests your patience. It had problems
sustaining a 100Mb/s NIC as FTP server. Replacing the system with a
cheap Sempron 140 K10 and 4GB RAM solved all my stability and
performance problems with ZFS and I never looked back.

----------------------------------------------------------------------
crest_ · 2016-02-23 12:04

The problem wasn't that there it lacked physical RAM but that it
lacked address space. There was enough swap available and even some
free RAM but the kernel heap was too fragmented. Simply put there was
no hole large enough to hold a 128KiB allocation inside the kernel
heap. If there is no address space to map the physical memory to than
it doesn't matter if there is any physical memory to map.

On i386 most hardware is restricted to 32 Bit in multiple ways. On the
one side i386 is limited to a 32 Bit linear address space (lets ignore
segmentation). PAE doesn't factor into it either because the hardware
had less than 4GiB physical memory. The kernel splits the address
space into 3GiB user space and 1GiB kernel space. Now the kernel has
to map devices into its 1GiB kernel address space. After this is done
the kernel memory map leaves about address space 300MiB to 400MiB to
the kernel heap.

Maybe the Raspberry Pi with fewer peripherals leaves enough for ZFS to
work (slightly) better on ARM than on i386.