Sustainably hosted, Managed F/OSS

Object storage, Redis, Prometheus

on Hetzner, OVH and LeaseWeb

**tl;dr - Line by line explanation of my ansible-powered ZFS install script for use on Hetzner’s dedicated hardware (Ubuntu 20.04 - “Focal”) – it’s not perfect/minimal, but it works for me

A while back I started using ZFS on all my bare metal dedicated hardware hosted at Hetzner to wrangle the attached HDDs and SSDs. There are lots of choices in the space (standard LVM,

mdraid

btrfs, etc), but I chose ZFS for it’s feaureset and ergonomics.

I won’t bore anyone with the why, but one of the things I needed to navigate was how to install newer versions of ZFS on Ubuntu 20.04 (one of the supported operating systems at Hetzner). I encountered some issues (particularly post-install) while setting up ZFS on my systems so I wanted to give a walkthrough of how I did it.

This post is a stanza-by-stanza explanation of my Ansible scripts that install ZFS.

Before you know if ZFS (or any filesystem) is worth switching to, you’re probably going to want to RTFM. ZFS has a lot of ins andand I certainly am not an expert in it, but knowing the how and why, as well as terminology and even history of the project is very important.

A few links to get you started:

Along with ZFS (which is likely the most unknown quantity here) you probably want to be familiar with Ubuntu and general systems administration. That’s quit a big area to cover but here are a few links:

It almost goes without saying but if you’re not very familiar with linux system administration at this point, you probably shouldn’t be attempting this.

I personally choose to have my disks in a somewhat custom configuration – I have RAID1 (mirroring) setup via

mdraid, along with partitions on each disk that I can give to ZFS to manage. Ideally, I’d have entire disks to give to ZFS but partitions works too.

OS setup (

installimage) which includes disk setup is run from Hetzner’s rescue mode and can be guided with a file like the following:


# Drive declarations used by installimage DRIVE0 /dev/nvme0n1 DRIVE1 /dev/nvme1n1 # Enable Software RAID (for the OS disk) SWRAID 1 SWRAIDLEVEL 1 # Bootloader (generally grub) BOOTLOADER grub # Machine hostame HOSTNAME machine01 # Partition configuration PART swap swap 32G PART /boot ext4 1G PART / ext4 128G # This last parittion will get made (wiped & recreated as ZFS later) PART /root-disk-remaining ext4 all # You can specify images that Hetzner uses by accessing their network share IMAGE /root/.oldroot/nfs/images/Ubuntu-2004-focal-64-minimal.tar.gz

This is obviously quite Hetzner-specific, but do whatever you need to do on your own systems to have partitions/drives available for ZFS to utilize.

zfsutils-linuxand

zfs-zed

In Ansible YAML the step looks like this:

- name: Hold all zfs-related upstream packages ansible.builtin.shell: | apt-mark hold zfsutils-linux apt-mark hold zfs-zed

As of the writing of this post the version of

zfs-linux in Ubuntu Focal is 0.8.3. Since our goal here is to install and keep using a newer version, we want to make sure that any

apt updates do not replace our installed version with an older version.

While we’re here, let’s purge the packages as well.

- name: Purge ZFS upstream (ubuntu) packages if installed ignore_errors: yes ansible.builtin.command: | apt purge --allow-change-held-packages zfsutils-linux zfs-zed

You can read the docs on

apt to see what

purge does – it’s similar to remove but also removes configuration files if present.

You can install the dependencies for OpenZFS like so:

- name: Install requirements for building ZFS ansible.builtin.apt: name:packagesupdate_cache: yes state: present vars: packages: - build-essential - autoconf - automake - libtool - gawk - alien - fakeroot - dkms - libblkid-dev - uuid-dev - libudev-dev - libssl-dev - zlib1g-dev - libaio-dev - libattr1-dev - libelf-dev # The line below calls for the output of `uname -r`, for example "5.16.5-arch1-1" for my arch system # so the line would resolve to something like "linux-headers-5.16.5-arch1-1" (not valid on Ubuntu of course, but as an example) - linux-headers uname_r.stdout }} - python3 - python3-dev - python3-setuptools - python3-cffi - libffi-dev - python3-packaging - git - libcurl4-openssl-dev

There may be a few dependencies that aren’t strictly necessary but just about all of them should be required.

/opt/zfs

Here is ZFS:

- name: Create /opt/zfs ansible.builtin.file: path: /opt/zfs state: directory

If you want to download ZFS:

- name: Download zfs ansible.builtin.get_url: url: "httpsgithub.com/openzfs/zfs/releases/download/zfs _zfs_version zfs _zfs_version tar.gz" checksum:_zfs_tarball_sha256_checksummode: 0755 dest: "/opt/zfs/zfs _zfs_version tar.gz"

Here the substitution

{{ _zfs_version }} (

2.1.1. You’ll also want to do the download yourself and collect/generate a checksum to use. Never download stuff from the internet that isn’t supposed to change without a checsum!

{{is the Ansible templating syntax) is


And if you want to copy it in from the computer where Ansible is running (this is what I do):

- name: Copy zfs package (avoid rate limit) ansible.builtin.copy: src: files/zfs/zfs _zfs_version tar.gz" mode: 0755 dest: "/opt/zfs/zfs _zfs_version tar.gz"

Regardless of how you choose to get the source you’ll need to unzip it of course:

- name: Unzip zfs code ansible.builtin.unarchive: src: "/opt/zfs/zfs _zfs_version tar.gz" dest: "/opt/zfs" remote_src: yes

To start the build process:

- name: Setup and Build ZFS ansible.builtin.shell: | ./autogen../configure make clean make -j args: chdir: "/opt/zfs/zfs _zfs_version 

If you’re familiar with building things from source, various projects use this common toolset (some variant of

make, autogen and configure scripts). As you might expect this might take a while so give it some time.

After building ZFS

- name: Install ZFS ansible.builtin.shell: | make install args: chdir: "/opt/zfs/zfs _zfs_version 

Normally you’d think we’d be done *right here*, but this post exists because doing this is generally not enough! Let’s press onwards.

First force the unload of the ZFS module(s) if they are running:

- name: Force unload of ZFS module(ss) ansible.builtin.shell: | ./scripts/zfs.-u args: chdir: "/opt/zfs/zfs _zfs_version 

It would be ideal to *not* be running any workloads when this is happening, as you might expect.

While the modules are unloaded, we can instlal some helpers:

- name: Post-Install ZFS Helpers ansible.builtin.shell: | ./scripts/zfs-helpers.-i args: chdir: "/opt/zfs/zfs _zfs_version 

After the helpers are installed, let’s force reload the ZFS module:

- name: Force reload of ZFS module ansible.builtin.shell: | ./scripts/zfs.args: chdir: "/opt/zfs/zfs _zfs_version 

I’ve found that building and using the

deb module (the format used by Debian for package installation) helped as well with getting the installation to stick and not be replaced by Ubuntu package defaults.

First build the ZFS deb package from the source code:

- name: Build ZFS deb package ansible.builtin.shell: | make deb args: chdir: "/opt/zfs/zfs _zfs_version 

And then install it

- name: Install ZFS deb packages ansible.builtin.shell: | yes | dpkg -i --force-overwrite deb apt install -f -y deb args: chdir: "/opt/zfs/zfs _zfs_version 


Theoretically this step *should not* be necessary (or should be used alone), as we’ve already run an install process, but I’ve found that when doing one or the other I’d have situations where restarts would prompt an older version of ZFS to be used (despite the

apt purge) – particularly at the kernel level.

modprobe

If you want to enable the ZFS module that was installed immediately, use

modprobe:

- name: Modprobe zfs module block: - name: Install zfs kernel module community.general.modprobe: name: zfs state: present

Installing ZFS means installing a kernel module, but since we haven’t quite baked it in via Dynamic Kernel Module Support, we need to enable locally installed kernel modules to be used:

- name: Ensure extra is in front ansible.builtin.lineinfile: path: /etc/modules-load.d/modules.conf regexp: '^search' line: "search extra updates ubuntu built-in" state: present

What this does is ensure that the file that manages the kernel module search path

/etc/modules-load.d/modules.conf has a line in it that has

search extra updatesspecified. Normally there is at least one line that starts with

search, and what we want to do is make sure

extra module locations are searched early in the process.

We’ll want to install via the DKMS subsystem:

- name: dkms install zfs ansible.builtin.shell: | dkms install zfs _zfs_version }} args: chdir: "/opt/zfs/zfs _zfs_version 

At this point, you should be able to check the current active ZFS version, and it should show you something like this:

root@machine01 ~ # zfs version zfs-2.1.1-1 zfs-kmod-2.1.1-1

There are a few SystemD units (as usual Digital Ocean has some great docs) that are created but need to be enabled for ZFS to use:

- name: Ensure zfs-related systemd units are enabled block: - ansible.builtin.systemd: name:itemstate: started enabled: yes loop: - zfs-import-cache.service - zfs-import.target - zfs-mount.service - zfs-share.service - zfs-zed.service - zfs-volume-wait.service - zfs.target

The variables being looped over are akin to running

systemd start  and

systemd enable .

The security-minded reader at home is no doubt cringing into the atmosphere by now, but making sure my kernel upgrades are manual is the only way I’ve found to ensure that the installed version of ZFS was not disabled/unexpectedly altered/broken by kernel upgrades:

- name: Hold all kernel upgrades to prevent custom built ZFS from doing fallback ansible.builtin.shell: | apt-mark hold linux-headers-generic apt-mark hold linux-image-generic apt-mark hold {{ uname_r.stdout }} # you'll need that `uname -r` output again here

I personally prefer to hold back any kernel upgrades and instead perform them at machine setup rather than during operation. DKMS *should* ensure that with the building of any new kernel the ZFS code is rebuilt but it’s been flaky in the past so I find it hard to trust.

Along with all the kernel changes we’ve made so far (good and questionable), one thing we’ll want to do is add the configuration necessary to ensure the kernel module is loaded:

- name: Ensure zfs kernel module comes up with next restart tags: [ "zfs:post-install:config" ] block: - name: Add zfs module load file ansible.builtin.template: src: templates/kernel-modules/zfs.conf.j2 dest: /etc/modules-load.d/zfs.conf owner: root group: root mode: 0644

Now that we have ZFS installed (restart once to check I’m going to leave the rest of the cluster setup to you. There are a *lot* of ways to use ZFS and setup zpools (RAID0/1/5/Z

Actually using ZFS properly is out of scope (I can only hope I’ve covered *installing* it properly, at least), but please refer to the usual manuals there to set up your ZFS pool and storage requirements. Trying to cover even the basics of how to setup ZFS for your drives once it’s been installed is certainly a lot of reading so we’ll leave it there for today.

This is the part where you “draw the rest of the owl” (sorry).

txg_timeout

While experimenting with some write modes for Postgres on ZFS, I looked into an optimization modes that involved reducing the

txg_timeout from it’s default of 5 seconds to 1 second. While I won’t get into the tradeoffs implied by that move (please read my post on Postgres + ZFS), here is the setup that I’ve used for modifying

txg_timeout per-machine (some machines might use the optimization some might not):

- name: Add systemd unit to support txg timeout customization tags: [ "zfs:post-install:config" ] block: - name: Set current ZFS txg_timeout (to 1) ansible.builtin.shell: | echo 1 > /sys/module/zfs/parameters/zfs_txg_timeout - name: install zfs-config-txg-timeout service ansible.builtin.template: src: templates/zfs-config-txg-timeout.service.j2 dest: /etc/systemd/system/zfs-config-txg-timeout.service owner: root group: root mode: 0644 - name: start & enable zfs-config-txg-timeout service ansible.builtin.systemd: name: zfs-config-txg-timeout.service state: started enabled: yes daemon_reload: yes

The template that goes with that looks like this:


[Unit] Description=Set ZFS txg_timeout After=zfs.target ConditionPathIsDirectory=/sys/module/zfs [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/bin/bash -c 'echo {{ _zfs_txg_timeout_seconds | default(zfs_txg_timeout_seconds) }} > /sys/module/zfs/parameters/zfs_txg_timeout' # Not needed since we want it to always be whatever the setting is # ExecStop=/usr/bin/bash -c 'echo {{ _zfs_txg_timeout_seconds | default(zfs_txg_timeout_seconds) }} > /sys/module/zfs/parameters/zfs_txg_timeout' [Install] WantedBy=multi-user.target

Obviously you’ll want to define

{{ _zfs_txg_timeout_seconds }} and the

zfs_txg_timeout_seconds. In general, this will make sure that the

txg_timeout is set to what you want it to be upon startup after ZFS has started.

Wasn’t that easy? If you’re answering no, it wasn’t easy for me to figure out either! I spent a lot of time wondering why DKMS wasn’t working properly (which is why the hack of preventing kernel upgrades is still included) and dealing with other issues. The OpenZFS documentation is pretty great but seems to be missing a guide somewhat like this, so hopefully this post will save people some time going forward when trying to experiment with new ZFS setups on Hetzner.

Another thing I’m hoping for with this post is to be corrected – if you see something glaringly wrong in my setup, please reach out.

The more people can use low cost infrastructure providers like Hetzner, the more cool software we get and the more interesting/innovative products can be built. To that end I’m working on NimbusWS (I’m behind on this but should launch by end of Q1 2022) – if you’re interested please sign up for the service and kick the tires (free tier usage will be available).