https://andreas.welcomes-you.com/zfs-risc-v-512mb-lichee-rv/

Andreas' website Menu

  * Home
  * Computer projects
  * Music
  * Links

[htop-on-randomFiles-write-to-zfs]
13 March 2022

ZFS on a single core RISC-V hardware with 512MB (Sipeed Lichee RV D1)

Is this feasible?

Yes it is! OpenZFS on Linux compiled from sources for RISC-V, it will
work with less than 250MB of RAM for a nearly fully filled 2TB zfs
volume with 225.000 files! 

Even stressing the system and run in parallel (via Wifi): a scp copy
(with 650kB/s), a samba transfer (with 1.3MB/s), writing random data
with dd on disk (with 2.8MB/s), finding files, running apt update,
zfs auto snapshot and doing a zpool scrub, the board needs well below
450MB of RAM.

[Overview-htop-iotop-etc-in-stress-test-thumbnail] Overview giving
info from htp, iotop and zfs

My conclusion

Linux and OpenZFS are an engineering marvel! And the OpenZFS
community has ported it successfully to RISC-V hardware, considering
in particular that there isn't a lot of hardware available yet. Well
done Linux and OpenZFS community!

...and the recommendation of 1GB of RAM per TB of storage?

I think I showed that having 256MB per TB is feasible -and probably
one can manage even more TB of storage with the same amount of
memory- hence you don't need 1GB of RAM per TB, but you might want to
have more for performance reasons.

[Update-2022-03-13] As I learned from Durval Menezes on the
zfsonlinux forum, the: "1GB RAM per 1TB Disk" rule used to be the
case until a few years ago (around ZFS 0.7.x-0.8.x IIRC), but then
our cherished devs did a round of specific memory optimizations and
since then it's no longer the case. [/Update]

 

The details

Below I have some more details on how I have configured the system,
including some tricks I had to apply to get ZFS running on limited
memory without getting a kernel panic, the performances I measured
and how I have build the ZFS kernel module and tools on the Sipeed
Lichee RV RISC-V board.

Performances

To estimate the maximum read/write speeds (in MB/s) I used 1GB
sequential reads/writes on the Sipeed Lichee RV carrier board with
RISC-V D1 processor. 1GB was chosen to ensure that any possible
caching will not have a big impact on the measurements (the board has
only 512MB of RAM). 

+---------------------------------------------------------+
|                     |Write|random data|Read |           |
|                     |zeros|           |zeros|random data|
|---------------------------------------------------------|
|256GB SSD via USB2                                       |
|---------------------------------------------------------|
|ext4                 |35.2 |12.4       |41.3 |40.9       |
|---------------------+-----+-----------+-----+-----------|
|zfs                  |38.2 |15.1       |42.0 |41.7       |
|---------------------+-----+-----------+-----+-----------|
|zfs native encryption|4.0  |3.0        |3.7  |3.7        |
|---------------------+-----+-----------+-----+-----------|
|LUKS + zfs           |8.0  |5.7        |7.6  |7.6        |
|---------------------------------------------------------|
|2TB HDD via USB2                                         |
|---------------------------------------------------------|
|zfs                  |22.5 |13.3       |27.1 |26.1       |
|---------------------------------------------------------|
|Internal SD card (MMC)                                   |
|---------------------------------------------------------|
|ext4                 |12.8 |10.4       |12.1 |12.1       |
+---------------------------------------------------------+

It is impressive that zfs is more performant than ext4. However, the
native encryption of ZFS lags much behind the combination LUKS+zfs.
This is not specific to the RISC-V implementation, from my experience
this is also the case on Intel architecture.   

 

The tricks

ZFS ARC size

The ZFS kernel modue by default allocates half of the system RAM for
caching/buffers. This would amounts to 256MB for the Lichee RV board.
One can query the min and max ZFS ARC sizes directly from the kernel
module 

cat /sys/module/zfs/parameters/zfs_arc_min
cat /sys/module/zfs/parameters/zfs_arc_max

or get a summary with the arc_summary tool

arc_summary -s arc

The sizes can be adapted, either by defining them in /etc/modprobe.d/
zfs.conf, or by updating them in the running system with  

sudo echo size_in_bytes >> /sys/module/zfs/parameters/zfs_arc_max

It seems that the lowest value one can set is 64MB (67108864 bytes).
Hence this is what I have used in the tests above.

Dropping caches

Reducing the ZFS caches helps already a lot, but to ensure that the
kernel memory will not get fragmented and as a consequence the system
runs out of memory, one can use a feature that tells the kernel to
drop the caches. A crontab did the job on the RISC-V board by having
the following line in the root crontab (dropping every 10min the
caches):

*/10 * * * * /usr/bin/sync; /usr/bin/echo 3 > /proc/sys/vm/drop_caches

This approach was needed to prevent kernel panics, before I discoved
that one can reduce the ZFS caches. But honestly it doesn't hurt -
except that one might get less performance. I didn't mind, since this
whole process was more to experiment than really setting up a NAS
based on RISC-V. Althought it is not impossible to use this board as
a NAS, if you don't mind maybe a bit lower transfer speeds than you
normally might expect (I get around 12MB/s via samba from an
unencrypted ZFS volume).

 

How to build OpenZFS on RISC-V

As basis I have used the boot software and Debian root file system
which I had documented in my other blog post.

On Debian unstable (at time of writing, March 2022: bookworm/sid),
OpenZFS is not yet available as debian package (zfsutils-linux).
Hence you have to compile and build OpenZFS yourself.

Unfortunately OpenZFS seems not to support (yet) cross-compiling for
the RISC-V platform, hence you have to build the kernel on the RISC-V
board. But since the ZFS kernel module has to be compiled with the
same compiler as the kernel -and I used gcc 9 to cross-compile the
kernel and gcc 11 is provided with Debian bookworm/sid- I had also to
recompile the Linux kernel on this single core RISC-V processor. This
took 4.5h, but ok, I really wanted to get ZFS running :-).  

To build, you should follow the instructions given by OpenZFS which
first tells you the packages that are needed

sudo apt install build-essential autoconf automake libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev

then you clone the repository

git clone https://github.com/openzfs/zfs.git

and execute

cd ./zfs
git checkout master
sh autogen.sh
./configure --with-linux=/your/kernel/directory
make -s -j$(nproc)

Finall you can install the kernel module

sudo make install; sudo ldconfig; sudo depmod

To test the module, you can run

modprobe zfs

and if successful, you can add a new line in /etc/modules with "zfs".
Done!

 

ZFS auto snapshot

Since the debian package zfs-auto-snapshot is not yet available on
RISC-V, probably due to its dependency to zfsutils-linux, I
downloaded it from zfsonlinux/zfs-auto-snapshot and followed the
instruction in the README:

wget https://github.com/zfsonlinux/zfs-auto-snapshot/archive/upstream/1.2.4.tar.gz
tar -xzf 1.2.4.tar.gz
cd zfs-auto-snapshot-upstream-1.2.4
make install

This will create backups every 15min (4x), hourly (24x), daily (31x),
weekly(8x) and monthly (12x).

This article was updated on 13 March 2022

  * risc-v
  * zfs

You should also read:

11 March 2022

Building boot software and Debian from sources for a RISC-V board
(Sipeed Lichee RV with D1 processor)

Copyright 2022 - Powered by Publii - Impressum