Cloud at home with minimum toil – PXE / Proxmox / Saltstack / k3s

My home lab was 7 years old and it was time to replace it. It was based on five Odroid HC1 nodes and 1 Odroid N1 (which never reached the mass production stage, Hardkernel sent it to me as a gift for a debug party). 4 HC1 nodes were used for a Docker Swarm cluster and 1 was dedicated to Nginx as a reverse proxy / WAF / SSL offloader. Regarding the Odroid N1, it was used as a NAS and also as a saltstack master.

Everything has been replaced with a major upgrade: faster infrastructure, better power efficiency, GitOps approach, virtualized environments, Kubernetes cluster, improved Saltstack implementation and last but not least: everything can be built automaticaly from scratch using PXE and the help of custom scripts  !

This is the (very long) story of my new homelab build…

Concept: the big picture

Before diving into all the details, I think it is important to understand what was goind through my mind, the concept I tried to implement (and hopefully succeeded in doing so).

I’m not saying that it is the best approach. After all, it’s a homelab,which, to me, means a way to learn, have fun, express creativity, and, of course, self-host some services. 

Simplified GitOps approach

I did not use any CI/CD components, no branches (at least for now), but the bare minimum for Infrastructure as Code is in place:

  • 100% “as code” infrastructure
  • Git repository
  • IaC frontend through Saltstack
  • IaC backend implementation with Proxmox VE

 

Infrastructure overview

A homelab comes with certain constraints that sometimes prevent you from making the best technical choices. For me, the main concerns were:

  • Money: Obviously, resources are limited and must be optimized based on what you want or need to achieve.
  • Space: Hardware takes up room, can be noisy, etc.
  • Power consumption: For hardware running 24/7, this can be a real game-changer for the annual electricity bill.
  • Ease of replacement: How easily things can be replaced in case of failure, as I don’t have tons of free time!

Hardware

In this homelab revision, I chose x86 hardware over ARM. The Intel N100 is a powerhouse in terms of performance, efficiency, and price. Not to mention that it offers far greater expansion options than ARM SBCs, especially when it comes to RAM and virtualization support.

Devices

Basically, my “cloud@home” setup consists of two Intel N100 nodes and one N6000 (Liva Z3). Since I have a small, wall-mounted 19″ rack with a depth of 30 cm, I decided to build custom servers using rack enclosures.

Below is the list of components I used:

Item Quantity Comments
Intertech 1.5U-1528L 1 Case for Node-1. A surprisingly good enclosure for the price: only 1.5U in height, with two externally accessible 3.5″ bays for SATA drives. It supports a Mini-ITX motherboard and has a depth of 27.8 cm.
Intertech IPC 1HU-K-125L 1 Case for Node-2. Similar to the previous one but with a 1U height and no externally accessible 3.5″ bay.
Fortron FSP250-50FEB 2 A 250W Flex ATX power supply with up to 85% efficiency—useful for a setup running 24/7.
ASUS Prime N100I-D D4-CSM 2 Mini-ITX motherboard with an integrated N100 processor. I found it to have a good balance of features and price.
32Gb Kingston FURY Impact, DDR4 3200 SO-DIMM 2 Contrary to the official specs, the N100 is not limited to 16GB of RAM—it works perfectly with 32GB.
Kingston NV2 – 1Tb 2 Used for virtual machines and container storage. Offers good performance and capacity for the price. More than enough for my home lab.
SATA SSD kingston kc600  256Gb 2 Boot disks and /root  storage for Proxmox on both Node-1 and Node-2.
Uni USB 3.0 to 2.5Gbps ethernet adatpter. 4 The ASUS motherboard has only a single gigabit NIC, which was too limiting for my needs. Adding two additional ports allows for bonding (LACP) plus a dedicated 2.5G node-to-node connection for the replication network and Corosync. I initially tried dual-port PCIe cards, but none worked properly (r8123, r8125, Intel 82571, and Intel i225-V).
JMB582 based SATA 3 M.2  controller with 2 ports (Key A + E) 1 Used to connect the two hard drives for the NAS VM. I could have used a PCIe card, but the only available slot was initially occupied by a network card… well, initially.
Seagate HDD 8Tb Sata 3 2 Used with the JMB582 controller for NAS storage.
Liva Z3 – 128 Gb 1 Used as Node-3 for Proxmox quorum, Salt Master, Zabbix monitoring, and a third Kubernetes node (mainly to improve etcd cluster stability).
Kingston fury DD4 3200 8Gb 2 16GB dual-channel RAM for the Liva Z3.
Patriot P400 Lite 250 Gb 1 Low-power PCIe M.2 SSD for the Liva Z3

Bios settings

I did some BIOS “tweaking” mostly to save power and to allow for PXE boot on all nodes (including wake-on-lan activation)

Asus Prime N100I-D

Here are the settings for the two N100 nodes:

  • dvmt graphic memory set from 64 to 32 : no video memory needed for a headless server
  • enabled sr-iov support for better offloading on NIC
  • disabled all usb port but usb_3, usb_4, u32g1_e3 and u32g1_e4 which are the one mounted on the face and the one used for the etherner adapters.
  • enabled network stack, ipv4 PXE and set PXE boot as first priority (can be done only after a reboot)
  • set restore AC power loss to last state in case of power failure, so it will be powered back on
  • enabed wake-on-lan (which is called “power on by pci-e”) so I could turn on my servers from pfsense easily
  • disabled hdaudio, not needed, save power
  • disabled wifi and bluetooth. Through, there are no such functionnalities installed
  • disabled serial and parallel ports, save power
  • enabled XMP as the memory I use support it. It might set timings to better ones, can’t hurt.
  • Disable fast boot : server’s are not willing to reboot a lot and I prefer letting all hardware checks to be done when I reboot it
  • enabled native aspm for power saving

Liva z3

  • Power management / resume via PME : mandatory to allow wake-on-lan
  • Disabled ACPI sleep state: I don’t need and don’t want it to be able to sleep as a 24/7 server 
  • Wireless function : disabled wifi and bluetooth, I do not need then and it can save power
  • System agent configuration: set all memory values to mininum as it will be a headless machine, no need for GPU memory
  • PCH configuration: disabled audio (not needed + power save), set restore AC power lost to power on
  • Boot / network stack: enabled (wol), disabled quiet boot and set boot order to  “usb, network, harddisk” 
  • disabled EUP else wake-on-lan won’t work

Photos

I took some photos during the “first” assembly process. There have been some change since I replaced the PCIe network cards by USB-3 adapters and finaly used 32Gb So-dimms.

Base infrastructure 

Prepare for PXE boot

As described previously, I configured all 3 nodes to boot via Network / PXE by default and fallback to internal storage. The main goal is to be able to “factory reset” any server while rebooting (all “post-install” configuration is then done by Saltstack).

If I want to restart one node from scratch, I just enable pxeboot on my PFsense box, set the corresponding boot file and reboot the device. Once the install has finished, I disable pxeboot and the device fallback to the media on which the fresh install has been done.

In Pfsense, I installed the package “tftpd”. then I configured it via the menu “Services / TFTPD Server”:

  • Enable it
  • Restrict to ip adresses of vlan with PXE enabled devices
  • Set to ipv4 only as I don’t use IPv6 for now
  • Open udp port 69 in the corresponding vlan
  • Download IPXE and and add files on the TFTP Server (http://boot.ipxe.org/undionly.kpxe and http://boot.ipxe.org/ipxe.efi). In my case, I used scp to copy files on my PFsense box.
  • Add an autoxec.ipxe file that will be launched by ipxe

Bellow is the my autoxec.ipxe file:

Note that calling “dhcp” in first place seems to be redundant as the Bios / EFI already got an ip address to load ipxe. But In fact, without calling “dhcp”, I had timeout while loading the (quite big) Proxmox initrd file…

Then I configured PFsense’s DHCP service to send required information for PXE boot to work:

  • Uncheck ignore bootp queries
  • set TFTP Server to the corresponding IP of the router (for each vlan on which tftpd should respond)
  • Enable network booting
  • Set the boot files : undionly.kpxe for BIOS and ipxe.efi for UEFI

In order to be able to control and customize any node on my network, each one has a specific “autoexec.ipxe” and “initrd” files that I just rename when I need to re-stage them.

I keep the default autoexec.ipxe with the “exit 1” command so by default, my servers boot to the next BIOS option. It is not mandatory but it makes the boot process faster when PXE is not enabled.

Now booting any node from my network is possible and simply controled via my PFSense box.

Proxmox nodes provisioning

PXE image creation

What I wanted was to boot any node from the Proxmox ISO in “auto install” mode as explained on the wiki page https://pve.proxmox.com/wiki/Automated_Installation. This implies to generate a custom ISO image with my own answers file, then to convert the ISO into a PXE bootable file with  https://github.com/morph027/pve-iso-2-pxe.

I also needed to add some customizations to both install process and first boot. 

What I added to the the install process are:

  • Patch Proxmox to allow installing on an emmc disk (for the Liva z3)
  • Add a custom network interfaces file to setup the network correctly (vlan, bond, etc.)
  • Add a custom rc.local file to further customize after the installation process

During the first boot, I wanted to execute some actions so each node could be ready and fully configured with very few manual actions. This the purpose of the custom rc.local:

  • Configure network interfaces with the file injected in the ISO image
  • Configure /dev/sda to be fully dedicated to proxmox root (other storages are on nvme)
  • Remove enterprise repository and enable the community one
  • Remove subscription nag uppon login
  • Install saltstack so nodes can be automaticaly configured after the first boot
  • Create a zfs pool on the nvme disk
  • Enable wake-on-lan
  • Create a first LXE container on node-3 to host the salt-master
  • Create the Proxmox cluster on node-1

Each node is fully configured at first boot either by rc.local file or by saltstack which runs highstate for any new node.

This leads me to the create two Github projects:

  • A public “generic” one that allows to generate PXE image with automated Proxmox install and custom files : https://github.com/jit06/pve-auto-pxe
  • A private “specific” one: which contains my own custom files and a small build system to create all needed files in a folder that can be mounted as “/config” for pve-auto-pxe.

Obviously, I cannot share the private repository as it contains some informations I don’t want to share, but basically this is what it does:

  • Create a build folder with hostname sub folders in it
  • If a hostname’s specific rc.local exists, merge it with the main rc.local
  • Generate a finalized rc.local and autoexec.pxe files for all hosts
  • Ask for, then inject root password in answer.toml files
  • Ask for, and inject saltstack and git passwords for the Liva z3 node

This build script generates a kernel and initrd files that can be copied to the TFTP server on my Pfsense. The initrd file is 1.6Go in size and PFSense does not allow to copy such big file from the web interface. It has to be copied though ssh / scp.

Customized Proxmox images

The Proxmox ISO image customization is here to set up things that are more or less hardware related or needed right after a clean install. Everything else is set and / or tuned with saltstack.

As explained earlier, the initial configuration is done via a custom rc.local script injected in the Proxmox ISO image. All nodes have the same rc.local base plus a specific one.

Below is the common rc.local. It is executed only once as it is replaced by a new one via saltstack as soon as the salt-minion is connected.

As seen in the first lines, a dedicated /etc/network/interfaces.install is moved to replace the original one. This specific file is copied during the customization of the Proxmox ISO image. This file is very important because it set up the whole network. It is pretty similar on both node: a bridge dedicated to VM and LXC, and another bridge dedicated to the replication (corosync, zfs, etc.).

The interfaces file is like the following:

The node-1 has a special treatment because it has NL drives for NAS storage. At this stage, the script assumes that NAS drives are already formated ( mkfs.ext4 -F -b 4096 /dev/sdx1) because in case of restaging, I dont want to risk any loss of data.

This first rc.local also creates the corosync cluster:

Finally, the node-3 has a special treatment too on its rc.local: as this is the host for saltstack master, a dedicated LXC is created to be able to initialize everything else when all nodes are ready, including PGP initilization needed to encrypt and decrypt secrets that are stored on my private saltstack git repository (more details on this subject later).

Words in upper case like “SALTMASTER_ROOT_PWD” are injected by my custom build script during the ISO creation. As all rc.local files got deleted after the first connection to the saltmaster, I do not consider that as a big security concern.

The git repository used is a private one, which serves for GitOps via saltstack (more on that later).

The only remaining manual operations are listed below. After these actions, everything is set up and ready (including a fully functionnal k3s cluster with apps, again: more on that later):

  • reboot all nodes to unsure that all configurations are taken into account
  • adding node-2 to the cluster : it needs root password thus I can’t provide it in rc.local
  • add the qdevice on node-3 :
  • apply saltstack map states (iac_backend.host-map.sls) : can’t do it before the cluster is created
  • set saltmaster not to auto accept minions (auto_accept: false)

Saltstack + Git as IAC frontend

Concept

As seen previously, the salt master is an LXC container that is automatically configured during the very first boot of node-3 (the Liva Z3). This makes it fully reproducible, eliminating the need to rely on backups. The goal is to quickly set up a salt master from scratch and use it to configure everything else, including the salt master itself.

Basically, the concept is based on the following principles:

  • A private Git repository contains all state and pillar definitions
  • The repository is cloned regularly on the salt master LXC.
  • Any file change in the cloned repository triggers a highstate application on all registered minions.
  • Any new minion automatically applies a highstate upon registration.
  • A highstate is periodically applied to all minions.

This way, SaltStack acts as an Infrastructure as Code (IaC) frontend via Git: any push triggers changes in the infrastructure, removing the need for manual shell commands or direct connections to any server—whether it is an LXC, a virtual machine, or a Proxmox node.

Additionally, any new server is automatically configured, allowing me to rebuild parts or even the entire infrastructure with a simple salt '*' state.apply command.

Implementation

To implement the GitOps approach, one possible solution could have been to use GitFS to host SaltStack’s files. However, since I do not plan to make changes directly on the salt master (which, in my opinion, is an anti-pattern), I adopted a KISS approach: : a simple scheduled git pull task combined with an inotify-based state to apply any modifications.

To achieve this, the salt master has a scheduled task that pulls the repository every 2 minutes and a reactor configuration together with an inotify beacon.

The repository pull state is like the following (url and credentials are stored on pillar values, more on that later):

The “reactor.conf” file defines a state to be executed on each detected change:

And here is the “handle_changed_states.sls” content which simply applies hightstate on all registered minions

As explained earlier, with such a simple mecanism, any push to the git repository triggers any infrastructure and configuration changes, no need to login into the saltmaster nor any server. Of course any error will also be deployed very fast : “with great power come great responsibilities” 🙂

Secured secrets

Even though my SaltStack repository is private, it is still hosted on external cloud servers. Since the entire architecture relies on secrets such as passwords or private keys, I needed to set up a secure way to store this kind of information.

My setup follows SaltStack’s approach, which consists of managing secrets with pillar values for the storage and GnuPG for encryption

As previously mentioned, the custom rc.local file of node-3, executed during the first boot, initializes the GnuPG environment and injects the key pair (which I store in a private local location).

Thus, the SaltMaster LXC contains everything needed to encrypt a new secret with a command like:

The output is an encrypted string that can be used as saltstack pillar:

Saltstack repository architecture

I tried to follow the best pratices : pillar contains variables and customized values while states are mostly generics and depend on pillar’s values.

Bellow is a commented overview of the saltstack directory structure. Each element is detailled later.

level 1 level 2 comments
pillar    
  k3s settings to deploy the k3s cluster
maps define all virtual machines and LXC specifications for all nodes
services settings for specific services like reverse proxy (nginx)
users defines users and groups that should exists or be deleted
zabbix settings dedicated to zabbix states
git.sls settings for my saltstack git repository
iac_backend.sls define sysctl values and custom scripts for all proxmox nodes 
kubeapps.sls define applications that must be deployed in the k3s cluster
mail.sls settings for e-mail account and aliases for servers to be able to send e-mails
top.sls  
salt    
  iac_backend dedicated to proxmox backend deployments
iac_frontend dedicated to saltstack and gitops 
kubeapps states that deploy applications in k3s
reactor deploy reactor configuration, essentialy for the gitops approach
services deploy services as defined in states
sysadmin states dedicated to apply a standard configuration all servers, being LXC or VM 
users ensure that users and groups exist or are absents
top.sls  

Proxmox nodes as IAC_backend

Proxmox customization

I did some adjustements on Proxmox nodes either to “optimize” things for my modest hardware, to reduce power consumption or simply to install my custom scripts

The table below show the optimization related settings.

What How
Allow swap usage only when less than 1% free RAM. This preserve my SSD set vm.swappiness to 1
disable ipv6 as I don’t use it set net.ipv6.conf.all.disable_ipv6 to 1
Ensure there is at least 256Mb free in order to always be able to execute sysadmin tools (ssh, screen, netstate, etc.) set vm.min_free_kbytes to 262144
Better I/O multitasking performances by limitting the size of writebacks (ram cache to disk) Tset vm.dirty_ratio to 20

These settings are applied through states and pillar values.

Pillar are defined like the follwing:

And states applied to Promox nodes (more on udev rules in the next chapter):

Power consumption

One major advantage of my old ARM-based home lab solution was its low power consumption: it was around 60W, including my 24-port switch.

Switching to x86 required some BIOS tuning (see the beginning of this article), even with low-power CPUs. Unfortunately, I wasn’t able to achieve a huge reduction in power consumption. The initial power draw of the N100 nodes was about 19 watts. After applying BIOS tweaks and Linux adjustments, I managed to reduce it to 17.5W.

Here are my “low-power” udev rules, installed in /etc/udev/rules.d/99-powermgmt.rules

With everything running, including the NL drives, ethernet switch, pfSense box, and rack cooling fans, I measured between 68W and 75W total. I found this reasonable: it’s a 25% increase in power consumption compared to my old ARM cluster, but with more than 25% performance gains. In the end, this translates to about 150 euros per year, which is much cheaper than a similar cloud service.

Desired state configuration

As mentioned earlier, I use Proxmox as my IAC backend. To have a system that allows defining LXC containers and virtual machines in a desired state configuration style, I had to write some scripts.

I should have used salt-cloud, but at the time of writing, the current Proxmox extension isn’t very useful: it lacks reliable error reporting and is not yet officially integrated.

Terraform or OpenTofu could have been good candidates, but they are too complex for a simple infrastructure like mine.

My approach is based on VMs, templates, and LXC definitions through pillar values, with deployment managed via states and custom scripts to handle both creation and modification.

I wrote four scripts for VM and LXC creation/update. Their outputs follow SaltStack’s stateful script requirements.

set_common.sh: Common tools and definitions for all scripts.

set_lxc.sh: handle LXC creation and update

set_templates.sh: handle Proxmox VM templates

set_vm.sh: handle virtual machines creation and update

Below is how I defined templates, LXC and VM as pillar values to create and/or update them:

Now, the part of the states that handle these pillar values:

User management

My user management needs are pretty basic: system accounts and Samba shares. A simpler alternative to LDAP for my home lab is using states and minion values to handle all operations (create, modify, delete) and synchronize them across all hosts.

Example of user definitions in the pillar:

States to handle users on all nodes:

Zabbix as monitoring solution

I chose Zabbix to monitor all components of my home cloud. I found it easy to set up, yet very powerful, with many useful monitors by default.

I created a single state for the installation process, both for the server and all agents, depending on pillar values. I also had to write a small script to handle PostgreSQL database creation.

Example of pillar values:

States that install Zabbix:

As you can see, these state definitions rely on some external files.

zabbix.conf.php : jinja template for zabbix configuration

zabbix_server.conf: jinja template for Zabbix server configuration

zabbix_agent2.conf: jinja template for all Zabbix agents configuration. Note that there is a subtle settings here. If the host use kubernetes persistent volumes (detected with ‘k3s-labels:pv’), all vfs triggers are disabled. I did that because else, Zabbix agent would have mounted the persistent volumes on all nodes at the same same, which would creates corruption on the filesystem (more on persistent volumes later).

Here is the custom script that handle PostgreSQL database. It it compliant with saltstack’s stateful script requirements:

Some tips I noted about Zabbix:

  • The default system locale must be set for the Zabbix configuration process to work.
  • On the Zabbix server, the agent config file must use the local IP address (server=127.0.0.1). The FQDN does not work (but it works perfectly fine on all other agents…).
  • Don’t forget to open the following ports: 10050 (server to agents) and 10051 (agents to server).

I wanted to automatically add all managed hosts to the server right after agent installations using saltext-zabbix, but at the time of writing, it wasn’t working with the current SaltStack version.

I had to add all hosts manually… All in all, Zabbix is probably the least automated part of my homelab. Even the database restore and configuration deployment are manual. I probably need to work on that, but I found Zabbix not to be at a “cloud-ready” level yet (no “as code” configuration).

Ntfy as notification solution

A good monitoring solution is of limited use without a reliable notification mechanism. This is where ntfy comes into play.

Ntfy is a simple, free way to send and receive notifications on your smartphone. The usage is straightforward: just push a string via HTTP to a dedicated channel, and you can instantly receive the string on your smartphone.

Of course, the default free and cloud-based approach has limited security protection, but it meets the requirements for a home lab.

To pair it with Zabbix, a specific “mediatype” must be imported. You can find this in the following GitHub repository: torgrimt/zabbix-ntfy: Mediatype to add support for ntfy.sh services.

However, I found that the configuration of Ntfy’s mediatype in Zabbix is not well explained: 

Field Comment
URL Must contain the URL to the Ntfy server without “http://” or “https://”. For example, “ntfy.sh” is valid, but “https://ntfy.sh” is not valid.
Password, Token and Username These fields must be empty if not needed. By “empty,” I mean nothing—remove even the macro reference if not used (even if the macro is empty).
Topic Must contain the topic you want to use on the Ntfy server without any “/” character.

Backup strategy

Because most things are defined “as code” and based on a desired state approach, my backup needs are lightweight and very targeted: “live” data.

I actually have three types of “live” data to back up:

  • User files, hosted on a NAS (more on that later).
  • Zabbix database: I haven’t yet found an elegant way to back up and restore automatically via states. For now, I do a simple “pg_dump” from time to time (I’m mostly interested in backing up the configuration more than data history).
  • Kubernetes persistent volumes: This is what I will detail below.

My goal was to follow a KISS approach to make it easy to recover data in any circumstance. I mean “agnostic”: even if I had to change the infrastructure logic, hardware, containers, hypervisor, operating system, etc.

To achieve this, I based my implementation on ZFS volumes (prefixed by “pv_” for “Persistent Volumes”):

  • Any “pv” is a ZFS volume created on all Proxmox nodes and mounted on all Kubernetes nodes.
  • “pv” volumes are synchronized using “pvesync” at the hypervisor level.
  • All “pv” volumes are backed up from node-1 (which has NL disks) by simply mounting them in read-only mode and creating a tar.gz archive.
  • I keep 7 rolling backups for each volume.

More details on how these persistent volumes are handled are in the K3s chapter below.

Regarding the synchronization of ZFS volumes, “pvesync” works with a master/slave logic. In other words, it doesn’t handle bi-directional synchronization. This can be a problem because Kubernetes pods can freely move from one node to another. So, it’s necessary to determine which host holds the “master” ZFS volume in order to initiate synchronization accordingly.

After some research, I found that the best way to determine the “master” volume from the hypervisor was by checking the “bytes written” property of the ZFS volume. I based the selection of the “master” volume on the node where the ZFS volume has the highest “bytes written” value.

Here is the script. It determines the master node and then launches the synchronization if needed:

This script runs every 5 minutes, which is acceptable for me as I have very few write operations on my containers. The probability of losing data because a pod might have been moved just after a write operation but before the next synchronization is very low.

Even though it works very well, such a mechanism must be properly monitored to quickly detect any errors (e.g., network failure). Each call of the script logs the results in a file ( /var/log/zfs_sync.log).

I created a dedicated Zabbix item and trigger to be sure I’m alerted by Ntfy in case of error:

After several months in production, I’ve never lost anything and haven’t experienced any filesystem corruption (I use ext4 on top of ZFS volumes). However, I did encounter some errors—not due to the mechanism itself, but because of the USB network adapter dedicated to the replication network on one of the nodes. This adapter is sometimes reset by the kernel.

Last but not least, persistent volumes are automatically created by states and defined in pillar values. I created another script to handle their creation. This script also takes care of automatically restoring any backup during the creation process. So, again, if I deleted everything, SaltStack will recreate all volumes with their latest available data.

Here is how I define a persistent volume in pillar:

How states handle the creation:

The script beneath the states (/usr/local/bin/set_zfspv.sh, which is statefull compliant):

You can also see a call to a script named “/usr/local/bin/nas.sh”, it will be detailled later.

Automatic updates

As of writing, all my LXC and virtual machines are based on Debian 12. Thus I deployed unnattended upgrade on all of hosts via a simple state, based on the official documentation:

However, I choosed not to apply automated upgrade on Proxmox nodes as it may cause serious issues due to breaking changes (eg. change in network cards names).

Real life use cases 

NAS

As explained previously, one of the nodes (node-1) has two 8-terabyte hard drives. The main purpose is to build a NAS, primarily to offer Samba shares that host music, movies, office files, backups, etc.

One disk is “live,” and the other is used solely as a mirror/backup. The mirror disk is switched off most of the time and is even removed when I leave home for more than two days.

The mirroring is handled on the hypervisor side with the help of a custom script. This script also manages ZFS persistent volume backups and restores, and it can generate email reports.

In additions, most important files are also synchronized on Onedrive which is monitored via a custom log item plus a trigger in Zabbix for the file “/var/log/onedrive”

The NAS service itself is provided by a dedicated LXC on node-1. This LXC uses the “online” hard disk to provides SMB shares on a VLAN dedicated to users. Users confguration has been covered in the previous chapter “Users management”.

The state that configure the LXC is pretty simple:

Reverse proxy and WAF

As explained later, I have a Kubernetes cluster on which some exposed services are running. These services are behind an Nginx server that serves as a reverse proxy, SSL offloading, and a web application firewall thanks to NAXSI.

Nginx is fully deployed on a dedicated LXC running on node-2 via a couple of states.

Below is an example of pillar values to set up Nginx. Crypto keys are encrypted with GPG: