Ngatur F/OSS kanthi lestari

Panyimpenan obyek, Redis, Prometheus

ing Hetzner, OVH lan LeaseWeb

**tl;dr - Panjelasan baris demi baris babagan skrip instalasi ZFS sing bisa digunakake kanggo digunakake ing hardware khusus Hetzner (Ubuntu 20.04 - âÂÂFocalâÂÂ) â  iku ora sampurna / minimal, nanging bisa kanggo kula

Sadurungé aku wiwit nggunakake ZFS ing kabeh hardware darmabakti logam Bare sandi tuan rumah ing Hetzner kanggo wrangle HDDs lan SSDs ditempelake. Ana akeh pilihan ing papan kasebut (LVM standar,

mdraid

btrfs, etc), nanging aku milih ZFS amarga iku feaureset lan ergonomics.

Aku ora bakal nglairake sapa wae, nanging salah sawijining perkara sing kudu dakgoleki yaiku carane nginstal versi ZFS sing luwih anyar ing Ubuntu 20.04 (salah sawijining sistem operasi sing didhukung ing Hetzner). Aku nemoni sawetara masalah (utamane sawise nginstal) nalika nyiyapake ZFS ing sistemku, mula aku pengin menehi pandhuan babagan carane aku nindakake.

Kiriman iki minangka panjelasan stanza-by-stanza saka skrip Ansible sing nginstal ZFS.

Sadurunge ngerti yen ZFS (utawa sistem file apa wae) kudu diganti, sampeyan bisa uga pengin RTFM. ZFS wis akeh in lan aku mesthi ora pakar ing, nanging ngerti carane lan apa, uga terminologi lan malah sajarah project penting banget.

Sawetara pranala kanggo miwiti:

Bebarengan karo ZFS (sing bisa uga jumlah sing paling ora dingerteni ing kene) sampeyan bisa uga pengin kenal karo Ubuntu lan administrasi sistem umum. Sing âÂÂs metu saka area gedhe kanggo nutupi nanging kene sawetara pranala:

Mesthi wae, nanging yen sampeyan ora ngerti babagan administrasi sistem linux ing wektu iki, mesthine sampeyan ora kudu nyoba iki.

Aku dhewe milih duwe disk ing konfigurasi sing rada khusus â Aku duwe RAID1 (mirroring) persiyapan liwat

mdraid, bebarengan karo partisi ing saben disk sing bisa menehi ZFS kanggo ngatur. Saenipun, aku duwe kabeh disk kanggo menehi ZFS nanging partisi uga bisa digunakake.

Setelan OS (

installimage) sing kalebu persiyapan disk sing ditindakake saka mode ngluwari Hetzner lan bisa dipandu nganggo file kaya ing ngisor iki:
# Deklarasi drive digunakake dening installimage DRIVE0 /dev/nvme0n1 DRIVE1 /dev/nvme1n1 # Aktifake Software RAID (kanggo disk OS) SWRAID 1 SWRAIDLEVEL 1 # Bootloader (umume grub) BOOTLOADER grub # Mesin hostame # Partisi mesin HOSTNAME01 konfigurasi PART swap swap 32G PART /boot ext4 1G PART / ext4 128G # Partisi pungkasan iki bakal digawe (dihapus& digawe maneh minangka ZFS mengko) PART / ROOT-disk-sisa ext4 kabeh # Sampeyan bisa nemtokake gambar sing digunakake Hetzner kanthi ngakses jaringan nuduhake IMAGE /root/.oldroot/nfs/images/Ubuntu-2004-focal-64-minimal.tar. gz

Iki temenan cukup Hetzner-tartamtu, nanging apa wae sing perlu kanggo nindakake ing sistem dhewe kanggo duwe partisi / drive kasedhiya kanggo ZFS nggunakke.

zfsutils-linuxand

zfs-zed

Ing Ansible YAML langkah katon kaya iki:

- jeneng: Tahan kabeh zfs-related paket hulu ansible.builtin.shell: | apt-mark terus zfsutils-linux apt-mark terus zfs-zed

Minangka nulis kirim iki versi saka

zfs-linux ing Ubuntu Focal yaiku 0.8.3. Amarga tujuan kita ing kene yaiku nginstal lan tetep nggunakake versi sing luwih anyar, kita pengin nggawe manawa ana

nganyari apt ora ngganti versi diinstal kita karo versi lawas.

Nalika kita kene, ayo ngresiki paket uga.

- jeneng: Purge ZFS hulu (ubuntu) paket yen diinstal ignore_errors: ya ansible.builtin.command: | apt purge --allow-change-held-packages zfsutils-linux zfs-zed

Sampeyan bisa maca docs ing

cocok kanggo ndeleng apa

purge â iku padha karo mbusak nanging uga mbusak file konfigurasi yen ana.

Sampeyan bisa nginstal dependensi kanggo OpenZFS kaya mangkene:

- jeneng: Instal syarat kanggo bangunan ZFS ansible.builtin.apt: jeneng: packagesupdate_cache: ya negara: saiki vars: paket: - mbangun-penting - autoconf - automake - libtool - gawk - alien - fakeroot - dkms - libblkid-dev - uuid -dev - libudev-dev - libssl-dev - zlib1g-dev - libaio-dev - libattr1-dev - libelf-dev # Baris ing ngisor iki njaluk output `uname -r`, contone "5.16.5-arch1- 1"kanggo sistem arch sandi # supaya baris bakal mutusake masalah kaya "linux-headers-5.16.5-arch1-1"(ora bener ing Ubuntu mesthi, nanging minangka conto) - linux-headers uname_r.stdout }} - python3 - python3-dev - python3-setuptools - python3-cffi - libffi-dev - python3-packaging - git - libcurl4-openssl-dev

Bisa uga ana sawetara dependensi sing ora perlu banget, nanging meh kabeh kudu dibutuhake.

/opt/zfs

Iki ZFS:

- jeneng: Nggawe /opt/zfs ansible.builtin.file: path: /opt/zfs state: direktori

Yen sampeyan pengin ngundhuh ZFS:

- jeneng: Download zfs ansible.builtin.get_url: url: "httpsgithub.com/openzfs/zfs/releases/download/zfs _zfs_version zfs _zfs_version tar.gz"checksum:_zfs_tarball_sha256_checksummode: 0755 dest_zfs/opt .gz"Ing kene substitusi

{{ _zfs_version }} (

2.1.1. Sampeyan uga pengin download dhewe lan ngumpulake / ngasilake checksum kanggo digunakake. Aja ndownload barang saka internet sing ora kudu diganti tanpa checsum!

{{yaiku sintaks template Ansible) yaiku


Lan yen sampeyan pengin nyalin saka komputer ing ngendi Ansible mlaku (iki sing dak lakoni):

- jeneng: Nyalin paket zfs (supaya watesan tarif) ansible.builtin.copy: src: files/zfs/zfs _zfs_version tar.gz"mode: 0755 dest: "/opt/zfs/zfs _zfs_version tar.gz"Preduli saka carane sampeyan milih kanggo njaluk sumber sampeyan kudu unzip iku mesthi:

- jeneng: Unzip kode zfs ansible.builtin.unarchive: src: "/opt/zfs/zfs _zfs_version tar.gz"dest: "/opt/zfs"remote_src: ya

Kanggo miwiti proses mbangun:

- jeneng: Setup lan Mbangun ZFS ansible.builtin.shell: | ./autogen../configure make clean make -j args: chdir: "/opt/zfs/zfs _zfs_version

Yen sampeyan wis ngerti mbangun barang saka sumber, macem-macem proyek nggunakake toolset umum iki (sawetara varian saka

nggawe, autogen lan ngatur skrip). Minangka sampeyan bisa nyana iki bisa njupuk sawetara wektu supaya menehi sawetara wektu.

Sawise mbangun ZFS

- jeneng: Instal ZFS ansible.builtin.shell: | nggawe nginstal args: chdir: "/opt/zfs/zfs _zfs_version

Biasane sampeyan bakal mikir yen bakal rampung * ing kene *, nanging kiriman iki ana amarga nindakake iki umume ora cukup! Ayo tekan terus.

Pisanan meksa mbongkar modul ZFS yen lagi mlaku:

- jeneng: Mbongkar paksa modul ZFS (ss) ansible.builtin.shell: | ./scripts/zfs.-u args: chdir: "/opt/zfs/zfs _zfs_version

Luwih becik *ora* nindakake beban kerja nalika kedadeyan kasebut, kaya sing dikarepake.

Nalika modul dibongkar, kita bisa nginstal sawetara asisten:

- jeneng: Post-Instal ZFS Helpers ansible.builtin.shell: | ./scripts/zfs-helpers.-i args: chdir: "/opt/zfs/zfs _zfs_version

Sawise helpers diinstal, ayo ngisi maneh modul ZFS:

- jeneng: Force reload modul ZFS ansible.builtin.shell: | ./scripts/zfs.args: chdir: "/opt/zfs/zfs _zfs_version

Aku wis nemokake bangunan kasebut lan nggunakake

modul deb (format sing digunakake dening Debian kanggo instalasi paket) uga mbantu supaya instalasi tetep tetep lan ora diganti karo standar paket Ubuntu.

Mbangun paket ZFS deb pisanan saka kode sumber:

- jeneng: Mbangun ZFS deb paket ansible.builtin.shell: | nggawe deb args: chdir: "/opt/zfs/zfs _zfs_version

Banjur nginstal

- jeneng: Instal ZFS deb paket ansible.builtin.shell: | inggih | dpkg -i --force-overwrite deb apt install -f -y deb args: chdir: "/opt/zfs/zfs _zfs_version
Secara teoritis, langkah iki * ora kudu * perlu (utawa kudu digunakake piyambak), amarga kita wis nglakoni proses instalasi, nanging aku nemokake yen nalika nindakake siji utawa liyane Âd duwe kahanan nalika miwiti maneh bakal njaluk versi lawas saka ZFS kanggo digunakake (sanajan

apt purge) â utamané ing tingkat kernel.

modprobe

Yen sampeyan pengin ngaktifake modul ZFS sing langsung diinstal, gunakake

modprobe:

- jeneng: Modprobe blok modul zfs: - jeneng: Instal modul kernel zfs community.general.modprobe: jeneng: negara zfs: saiki

Nginstal ZFS tegese nginstall modul kernel, nanging amarga kita durung cukup panggang liwat Dhukungan Modul Kernel Dinamis, kita kudu ngaktifake modul kernel sing diinstal sacara lokal supaya bisa digunakake:

- jeneng: Priksa manawa ekstra ana ing ngarep ansible.builtin.lineinfile: path: /etc/modules-load.d/modules.conf regexp: '^search'line: "nelusuri nganyari tambahan ubuntu built-in"negara: saiki

Apa iki mesthekake yen file sing ngatur path search modul kernel

/etc/modules-load.d/modules.conf wis baris ing sing wis

goleki nganyari tambahan sing ditemtokake. Biasane ana paling ora siji baris sing diwiwiti

telusuran, lan apa sing arep kita lakoni yaiku priksa

lokasi modul ekstra digoleki ing awal proses.

Kita pengin nginstal liwat subsistem DKMS:

- jeneng: dkms nginstal zfs ansible.builtin.shell: | dkms nginstal zfs _zfs_version }} args: chdir: "/opt/zfs/zfs _zfs_version

Ing jalur iki, sampeyan kudu bisa mriksa versi ZFS aktif saiki, lan kudu nuduhake sampeyan kaya iki:

root@machine01 ~ # zfs versi zfs-2.1.1-1 zfs-kmod-2.1.1-1

Ana sawetara unit SystemD (kaya biasane Digital Ocean duwe sawetara dokumen sing apik) sing digawe nanging kudu diaktifake supaya ZFS bisa digunakake:

- jeneng: Priksa manawa unit systemd sing gegandhengan karo zfs diaktifake blok: - ansible.builtin.systemd: jeneng:itemstate: wiwit diaktifake: ya loop: - zfs-import-cache.service - zfs-import.target - zfs-mount.service - zfs-share.service - zfs-zed.service - zfs-volume-wait.service - zfs.target

Variabel sing diubengi meh padha karo mlaku

systemd start and

systemd enable .

The security-minded reader at home is no doubt cringing into the atmosphere by now, but making sure my kernel upgrades are manual is the only way I’ve found to ensure that the installed version of ZFS was not disabled/unexpectedly altered/broken by kernel upgrades:

- name: Hold all kernel upgrades to prevent custom built ZFS from doing fallback ansible.builtin.shell: | apt-mark hold linux-headers-generic apt-mark hold linux-image-generic apt-mark hold {{ uname_r.stdout }} # you'll need that `uname -r` output again here

I personally prefer to hold back any kernel upgrades and instead perform them at machine setup rather than during operation. DKMS *should* ensure that with the building of any new kernel the ZFS code is rebuilt but it’s been flaky in the past so I find it hard to trust.

Along with all the kernel changes we’ve made so far (good and questionable), one thing we’ll want to do is add the configuration necessary to ensure the kernel module is loaded:

- name: Ensure zfs kernel module comes up with next restart tags: [ "zfs:post-install:config" ] block: - name: Add zfs module load file ansible.builtin.template: src: templates/kernel-modules/zfs.conf.j2 dest: /etc/modules-load.d/zfs.conf owner: root group: root mode: 0644

Now that we have ZFS installed (restart once to check I’m going to leave the rest of the cluster setup to you. There are a *lot* of ways to use ZFS and setup zpools (RAID0/1/5/Z

Actually using ZFS properly is out of scope (I can only hope I’ve covered *installing* it properly, at least), but please refer to the usual manuals there to set up your ZFS pool and storage requirements. Trying to cover even the basics of how to setup ZFS for your drives once it’s been installed is certainly a lot of reading so we’ll leave it there for today.

This is the part where you “draw the rest of the owl” (sorry).

txg_timeout

While experimenting with some write modes for Postgres on ZFS, I looked into an optimization modes that involved reducing the

txg_timeout from it’s default of 5 seconds to 1 second. While I won’t get into the tradeoffs implied by that move (please read my post on Postgres + ZFS), here is the setup that I’ve used for modifying

txg_timeout per-machine (some machines might use the optimization some might not):

- name: Add systemd unit to support txg timeout customization tags: [ "zfs:post-install:config" ] block: - name: Set current ZFS txg_timeout (to 1) ansible.builtin.shell: | echo 1 > /sys/module/zfs/parameters/zfs_txg_timeout - name: install zfs-config-txg-timeout service ansible.builtin.template: src: templates/zfs-config-txg-timeout.service.j2 dest: /etc/systemd/system/zfs-config-txg-timeout.service owner: root group: root mode: 0644 - name: start & enable zfs-config-txg-timeout service ansible.builtin.systemd: name: zfs-config-txg-timeout.service state: started enabled: yes daemon_reload: yes

The template that goes with that looks like this:


[Unit] Description=Set ZFS txg_timeout After=zfs.target ConditionPathIsDirectory=/sys/module/zfs [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/bin/bash -c 'echo {{ _zfs_txg_timeout_seconds | default(zfs_txg_timeout_seconds) }} > /sys/module/zfs/parameters/zfs_txg_timeout' # Not needed since we want it to always be whatever the setting is # ExecStop=/usr/bin/bash -c 'echo {{ _zfs_txg_timeout_seconds | default(zfs_txg_timeout_seconds) }} > /sys/module/zfs/parameters/zfs_txg_timeout' [Install] WantedBy=multi-user.target

Obviously you’ll want to define

{{ _zfs_txg_timeout_seconds }} and the

zfs_txg_timeout_seconds. In general, this will make sure that the

txg_timeout is set to what you want it to be upon startup after ZFS has started.

Wasn’t that easy? If you’re answering no, it wasn’t easy for me to figure out either! I spent a lot of time wondering why DKMS wasn’t working properly (which is why the hack of preventing kernel upgrades is still included) and dealing with other issues. The OpenZFS documentation is pretty great but seems to be missing a guide somewhat like this, so hopefully this post will save people some time going forward when trying to experiment with new ZFS setups on Hetzner.

Another thing I’m hoping for with this post is to be corrected – if you see something glaringly wrong in my setup, please reach out.

The more people can use low cost infrastructure providers like Hetzner, the more cool software we get and the more interesting/innovative products can be built. To that end I’m working on NimbusWS (I’m behind on this but should launch by end of Q1 2022) – if you’re interested please sign up for the service and kick the tires (free tier usage will be available).