Kusingathwa ngokuqhubekayo, Kuphethwe i-F/OSS
Isitoreji sento, Redis, Prometheus
ku-Hetzner, OVH kanye ne-LeaseWeb
**tl;dr - Incazelo yomugqa ngomugqa yeskripthi sami sokufaka se-ZFS esinamandla aphathekayo ukuze sisetshenziswe ku-HetznerâÂÂs ezinikezele zehadiwe (Ubuntu 20.04 - âÂÂFocalâÂÂ) â  ayilungile/incane, kodwa ingisebenzela
Ngemva kwesikhashana ngaqala ukusebenzisa i-ZFS kuwo wonke ama-hardware ami angenalutho ensimbi aphethwe e-Hetzner ukuze ngixabane nama-HDD nama-SSD anamathiselwe. Kunezinketho eziningi esikhaleni (i-LVM ejwayelekile,
mdraid
btrfs, njll), kodwa ngikhethe i-ZFS ye-feaureset ne-ergonomics yayo.
Ngeke ngikhathale noma ubani ngokuthi kungani, kodwa enye yezinto ebengidinga ukuyizulazula ukuthi ngizifaka kanjani izinguqulo ezintsha ze-ZFS ku-Ubuntu 20.04 (enye yezinhlelo zokusebenza ezisekelwayo e-Hetzner). Ngihlangabezane nezinkinga ezithile (ikakhulukazi ngemuva kokufaka) ngenkathi ngisetha i-ZFS kumasistimu ami ngakho-ke bengifuna ukunikeza indlela engikwenze ngayo.
Lokhu okuthunyelwe kuyincazelo ye-stanza-by-stanza yemibhalo yami ye-Ansible efaka i-ZFS.
Ngaphambi kokuthi wazi ukuthi i-ZFS (noma yiluphi uhlelo lwefayela) kufanelekile ukushintshela kulo, cishe uzofuna i-RTFM. I-ZFS inama-ins amaningi futhi futhi angiyena uchwepheshe kuyo, kodwa ukwazi ukuthi kanjani futhi kungani, kanye namagama kanye nomlando wephrojekthi kubaluleke kakhulu.
Izixhumanisi ezimbalwa zokukuqalisa:
Kanye ne-ZFS (okungenzeka ukuthi iyinani elingaziwa kakhulu lapha) cishe ufuna ukujwayelana no-Ubuntu kanye nokuphathwa kwezinhlelo ezijwayelekile. Lokho kushiya indawo enkulu okumele bayihlanganise kodwa nazi izixhumanisi ezimbalwa:
Kucishe kuhambe ngaphandle kokusho kodwa uma ungajwayelene kakhulu nokuphathwa kwesistimu ye-linux kuleli qophelo, cishe akufanele ukuthi uzame lokhu.
Mina ngokwami ngikhetha ukuba namadiski ami ekucushweni ngokwezifiso okuthi â  Nginokusethwa kwe-RAID1 (izibuko) nge-
mdraid, kanye nokuhlukaniswa kwediski ngayinye engingayinika i-ZFS ukuphatha. Ngokufanelekile, i-IâÂÂd inamadiski aphelele engingawanikeza i-ZFS kodwa ama-partitions nawo ayasebenza.
Ukusetha i-OS (
installimage) okuhlanganisa ukusethwa kwediski kuqaliswa kumodi yokuhlenga ye-HetznerâÂÂs futhi ingaqondiswa ngefayela elifana naleli elilandelayo:
# Izimemezelo zeDrayivu ezisetshenziswe ngokufaka isithombe DRIVE0 /dev/nvme0n1 DRIVE1 /dev/nvme1n1 # Nika amandla i-Software RAID (ye-OS disk) SWRAID 1 SWRAIDLEVEL 1 # I-Bootloader (ngokuvamile i-grub) BOOTLOADER grub # Umshini womshini01 Ingxenye HOSTNAME ukucushwa kwe-PART swap swap 32G PART /boot ext4 1G PART / ext4 128G # Lokhu kuhlukaniswa kokugcina kuzokwenziwa (kwesulwa& yakhiwe kabusha njenge-ZFS kamuva) INGXENYE /izimpande-idiski-esele ext4 konke # Ungacacisa izithombe ezisetshenziswa uHetzner ngokufinyelela ukwabelana kwenethiwekhi yabo IMAGE /root/.oldroot/nfs/images/Ubuntu-2004-focal-64-minimal.tar. gz
Lokhu kusobala impela ku-Hetzner-specific, kodwa yenza noma yini okudingeka uyenze kumasistimu akho ukuze ube ne-partitions/drive etholakalayo ukuze i-ZFS iyisebenzise.
zfsutils-linuxand
zfs-zed
Ku-Ansible YAML isinyathelo sibukeka kanje:
- igama: Bamba wonke amaphakheji ahlobene ne-zfs akhuphuka nomfula ansible.builtin.shell: | i-apt-mark bamba i-zfsutils-linux i-apt-mark bamba i-zfs-zed
Njengoba kubhalwa leli posi inguqulo ye
zfs-linux ku-Ubuntu Focal ingu-0.8.3. Njengoba umgomo wethu lapha uwukufaka nokuqhubeka nokusebenzisa inguqulo entsha, sifuna ukwenza isiqiniseko sokuthi noma iyiphi
izibuyekezo ze-apt azifaki esikhundleni inguqulo yethu efakiwe ngenguqulo endala.
Ngenkathi silapha, asihlanze namaphakheji.
- igama: Hlanza amaphakheji e-ZFS akhuphuka nomfula (ubuntu) uma efakiwe amaphutha_angaziwa: yebo ansible.builtin.command: | apt purge --vumela-change-holded-packages zfsutils-linux zfs-zed
Ungafunda amadokhumenti kuvuliwe
uyabona ukuthi yini
purge ingabe âÂÂifana nokukhipha kodwa iphinde isuse amafayela okulungiselela uma ekhona.
Ungafaka ukuncika kwe-OpenZFS kanje:
- igama: Izidingo zokufaka zokwakha i-ZFS ansible.builtin.apt: igama:packagesupdate_cache: yebo isimo: ama-vars akhona: amaphakheji: - build-essential - autoconf - automake - libtool - gawk - alien - fakeroot - dkms - libblkid-dev - uuid -dev - libudev-dev - libssl-dev - zlib1g-dev - libaio-dev - libattr1-dev - libelf-dev # Umugqa ongezansi ubiza okukhiphayo kokuthi `uname -r`, isibonelo "5.16.5-arch1- 1"yesistimu yami ye-arch # ukuze ulayini uxazulule kokuthile okufana ne-"linux-headers-5.16.5-arch1-1"(ayivumelekile ku-Ubuntu kunjalo, kodwa njengesibonelo) - i-linux-headers uname_r.stdout }} - python3 - python3-dev - python3-setuptools - python3-cffi - libffi-dev - python3-packaging - git - libcurl4-openssl-dev
Kungase kube khona ukuncika okumbalwa okungadingeki ngempela kodwa cishe konke okufanele kudingeke.
/opt/zfs
Nansi i-ZFS:
- igama: Dala /opt/zfs ansible.builtin.file: indlela: /opt/zfs state: directory
Uma ufuna ukulanda i-ZFS:
- igama: Landa i-zfs ansible.builtin.get_url: url: "httpsgithub.com/openzfs/zfs/releases/download/zfs _zfs_version zfs _zfs_version tar.gz"checksum:_zfs_tarball_sha256_sversion: .gz"Lapha esikhundleni
{{ _zfs_version }} (
2.1.1. Uzofuna futhi ukuzidawuniloda ngokwakho futhi uqoqe/ukhiqize isheke ongalisebenzisa. Ungalokothi ulande izinto ku-inthanethi okungafanele zishintshe ngaphandle kwe-checsum!
{{is the Ansible templating syntax) is
Futhi uma ufuna ukuyikopisha kukhompuyutha lapho i-Ansible isebenza khona (yilokhu engikwenzayo):
- igama: Kopisha iphakheji ye-zfs (gwema umkhawulo wesilinganiso) ansible.builtin.copy: src: amafayela/zfs/zfs _zfs_version tar.gz"imodi: 0755 dest: "/opt/zfs/zfs _zfs_version tar.gz"Kungakhathaliseki ukuthi ukhetha kanjani ukuthola umthombo uzodinga ukuwuvula nakanjani:
- igama: Unzip zfs code ansible.builtin.unarchive: src: "/opt/zfs/zfs _zfs_version tar.gz"dest: "/opt/zfs"remote_src: yebo
Ukuze uqale inqubo yokwakha:
- Igama: Setha futhi Yakha i-ZFS ansible.builtin.shell: | ./autogen../configure make clean make -j args: chdir: "/opt/zfs/zfs _zfs_version
Uma ujwayelene nokwakha izinto kusuka emthonjeni, amaphrojekthi ahlukahlukene asebenzisa le thuluzi elivamile (okunye okuhlukile kwe
yenza, i-autogen futhi ulungiselele imibhalo). Njengoba ungalindela ukuthi lokhu kungathatha isikhashana ngakho-ke kunikeze isikhathi.
Ngemva kokwakha iZFS
- Igama: Faka i-ZFS ansible.builtin.shell: | yenza ukufaka ama-args: chdir: "/opt/zfs/zfs _zfs_version
Ngokujwayelekile ungacabanga ukuthi sizokwenziwa *khona lapha*, kodwa lokhu okuthunyelwe kukhona ngoba ukwenza lokhu ngokuvamile akwanele! Masiqhubekele phambili.
Qala ngokuphoqelela ukulayishwa kwemojula ye-ZFS uma isebenza:
- igama: Phoqelela ukulayishwa kwe-ZFS module(ss) ansible.builtin.shell: | ./scripts/zfs.-u args: chdir: "/opt/zfs/zfs _zfs_version
Kungaba kuhle ukuthi *ungasebenzisi* noma yimiphi imithwalo yomsebenzi uma lokhu kwenzeka, njengoba ungalindela.
Ngenkathi amamojula elayishwa, singafaka abasizi abathile:
- igama: Thumela-Faka Abasizi be-ZFS ansible.builtin.shell: | ./scripts/zfs-helpers.-i args: chdir: "/opt/zfs/zfs _zfs_version
Ngemuva kokufakwa kwabasizi, ake siphoqelele ukulayisha kabusha imojula ye-ZFS:
- igama: Phoqelela ukulayishwa kabusha kwemojuli ye-ZFS ansible.builtin.shell: | ./scripts/zfs.args: chdir: "/opt/zfs/zfs _zfs_version
Ngithole lelo bhilidi nokusebenzisa i-
deb (ifomethi esetshenziswa i-Debian ekufakweni kwephakheji) isize futhi ngokwenza ukufakwa kunamathele futhi kungathathelwa indawo okuzenzakalelayo kwephakheji ye-Ubuntu.
Okokuqala yakha iphakethe le-ZFS deb kusuka kukhodi yomthombo:
- igama: Yakha iphakheji ye-deb ye-ZFS ansible.builtin.shell: | yenza i-deb args: chdir: "/opt/zfs/zfs _zfs_version
Bese uyayifaka
- igama: Faka amaphakheji we-ZFS deb ansible.builtin.shell: | yebo | dpkg -i --force-overwrite deb apt install -f -y deb args: chdir: "/opt/zfs/zfs _zfs_version
Ngokombono lesi sinyathelo *akumele* sidingeke (noma sisetshenziswe sodwa), njengoba sesivele senza inqubo yokufaka, kodwa ngithole ukuthi uma ngenza eyodwa noma enye Iâ Âd unezimo lapho ukuqalisa kabusha kungadala ukuthi kusetshenziswe inguqulo endala ye-ZFS (naphezu kokuthi
apt purge) â ikakhulukazi ezingeni le-kernel.
i-modprobe
Uma ufuna ukunika amandla imojula ye-ZFS efakwe ngokushesha, sebenzisa
I-modprobe:
- igama: Modprobe zfs module block: - igama: Faka i-zfs kernel module community.general.modprobe: igama: zfs state: present
Ukufaka i-ZFS kusho ukufaka imojuli ye-kernel, kodwa njengoba singakayibhaki nge-Dynamic Kernel Module Support, sidinga ukunika amandla amamojula afakwe endaweni ukuze asetshenziswe:
- igama: Qinisekisa ukuthi okungeziwe kungaphambili ansible.builtin.lineinfile: indlela: /etc/modules-load.d/modules.conf regexp: '^search'umugqa: "sesha izibuyekezo ezengeziwe ubuntu eyakhelwe ngaphakathi"isimo: samanje
Lokhu okwenziwayo ukuqinisekisa ukuthi ifayela eliphethe indlela yokusesha yemojuli ye-kernel
/etc/modules-load.d/modules.conf inomugqa kuwo onomugqa
sesha izibuyekezo ezengeziwe ezicacisiwe. Ngokujwayelekile kukhona okungenani umugqa owodwa oqala ngawo
ukucinga, futhi esifuna ukukwenza ukwenza isiqiniseko
izindawo zemojuli eyengeziwe ziseshwa ekuqaleni kwenqubo.
Sizofuna ukufaka ngesistimu engaphansi ye-DKMS:
- igama: dkms faka i-zfs ansible.builtin.shell: | I-dkms faka i-zfs _zfs_version }} args: chdir: "/opt/zfs/zfs _zfs_version
Kuleli qophelo, kufanele ukwazi ukubheka inguqulo yamanje ye-ZFS esebenzayo, futhi kufanele ikubonise into enjengale:
impande@machine01 ~ # zfs inguqulo zfs-2.1.1-1 zfs-kmod-2.1.1-1
Kunamayunithi ambalwa e-SystemD (njengenjwayelo i-Digital Ocean inamanye amadokhumenti amahle) adaliwe kodwa adinga ukunikwa amandla ukuze i-ZFS iwasebenzise:
- igama: Qinisekisa ukuthi amayunithi esistimu ahlobene ne-zfs anikwe amandla ibhulokhi: - ansible.builtin.systemd: igama:itemstate: iqalile inikwe amandla: yebo loop: - zfs-import-cache.service - zfs-import.target - zfs-mount.service - zfs-share.service - zfs-zed.service - zfs-volume-wait.service - zfs.target
Okuguquguqukayo okugoqiwe kufana nokusebenza
systemd start and
systemd enable .
The security-minded reader at home is no doubt cringing into the atmosphere by now, but making sure my kernel upgrades are manual is the only way IâÂÂve found to ensure that the installed version of ZFS was not disabled/unexpectedly altered/broken by kernel upgrades:
- name: Hold all kernel upgrades to prevent custom built ZFS from doing fallback ansible.builtin.shell: | apt-mark hold linux-headers-generic apt-mark hold linux-image-generic apt-mark hold {{ uname_r.stdout }} # you'll need that `uname -r` output again here
I personally prefer to hold back any kernel upgrades and instead perform them at machine setup rather than during operation. DKMS *should* ensure that with the building of any new kernel the ZFS code is rebuilt but itâÂÂs been flaky in the past so I find it hard to trust.
Along with all the kernel changes weâÂÂve made so far (good and questionable), one thing weâÂÂll want to do is add the configuration necessary to ensure the kernel module is loaded:
- name: Ensure zfs kernel module comes up with next restart tags: [ "zfs:post-install:config" ] block: - name: Add zfs module load file ansible.builtin.template: src: templates/kernel-modules/zfs.conf.j2 dest: /etc/modules-load.d/zfs.conf owner: root group: root mode: 0644
Now that we have ZFS installed (restart once to check IâÂÂm going to leave the rest of the cluster setup to you. There are a *lot* of ways to use ZFS and setup zpools (RAID0/1/5/Z
Actually using ZFS properly is out of scope (I can only hope IâÂÂve covered *installing* it properly, at least), but please refer to the usual manuals there to set up your ZFS pool and storage requirements. Trying to cover even the basics of how to setup ZFS for your drives once itâÂÂs been installed is certainly a lot of reading so weâÂÂll leave it there for today.
This is the part where you âÂÂdraw the rest of the owlâ (sorry).
txg_timeout
While experimenting with some write modes for Postgres on ZFS, I looked into an optimization modes that involved reducing the
txg_timeout from itâÂÂs default of 5 seconds to 1 second. While I wonâÂÂt get into the tradeoffs implied by that move (please read my post on Postgres + ZFS), here is the setup that IâÂÂve used for modifying
txg_timeout per-machine (some machines might use the optimization some might not):
- name: Add systemd unit to support txg timeout customization tags: [ "zfs:post-install:config" ] block: - name: Set current ZFS txg_timeout (to 1) ansible.builtin.shell: | echo 1 > /sys/module/zfs/parameters/zfs_txg_timeout - name: install zfs-config-txg-timeout service ansible.builtin.template: src: templates/zfs-config-txg-timeout.service.j2 dest: /etc/systemd/system/zfs-config-txg-timeout.service owner: root group: root mode: 0644 - name: start & enable zfs-config-txg-timeout service ansible.builtin.systemd: name: zfs-config-txg-timeout.service state: started enabled: yes daemon_reload: yes
The template that goes with that looks like this:
[Unit] Description=Set ZFS txg_timeout After=zfs.target ConditionPathIsDirectory=/sys/module/zfs [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/bin/bash -c 'echo {{ _zfs_txg_timeout_seconds | default(zfs_txg_timeout_seconds) }} > /sys/module/zfs/parameters/zfs_txg_timeout' # Not needed since we want it to always be whatever the setting is # ExecStop=/usr/bin/bash -c 'echo {{ _zfs_txg_timeout_seconds | default(zfs_txg_timeout_seconds) }} > /sys/module/zfs/parameters/zfs_txg_timeout' [Install] WantedBy=multi-user.target
Obviously youâÂÂll want to define
{{ _zfs_txg_timeout_seconds }} and the
zfs_txg_timeout_seconds. In general, this will make sure that the
txg_timeout is set to what you want it to be upon startup after ZFS has started.
WasnâÂÂt that easy? If youâÂÂre answering no, it wasnâÂÂt easy for me to figure out either! I spent a lot of time wondering why DKMS wasnâÂÂt working properly (which is why the hack of preventing kernel upgrades is still included) and dealing with other issues. The OpenZFS documentation is pretty great but seems to be missing a guide somewhat like this, so hopefully this post will save people some time going forward when trying to experiment with new ZFS setups on Hetzner.
Another thing IâÂÂm hoping for with this post is to be corrected â if you see something glaringly wrong in my setup, please reach out.
The more people can use low cost infrastructure providers like Hetzner, the more cool software we get and the more interesting/innovative products can be built. To that end IâÂÂm working on NimbusWS (IâÂÂm behind on this but should launch by end of Q1 2022) â if youâÂÂre interested please sign up for the service and kick the tires (free tier usage will be available).
