Why my servers run NixOS

Heads up: I’ve paused NixOS in my homelab for the moment, so this post isn’t fully up to date. astrid is powered off right now, and freyja has the GPU slice the two of them share. Everything below is how it ran while it was up, and I’ll update this if NixOS comes back.

I didn’t start on NixOS. I started on Windows, like most people, and after one too many broken installs I decided I needed a change. A friend of mine daily drove Arch at the time, and between him and a lot of Eric Murphy videos I talked myself into it. Arch felt like a challenge, and I genuinely wanted to understand what made it different from everything else I’d used.

Arch taught me a lot. It also taught me that I am really bad at remembering what I did to my own machine six months ago.

The problem with “I’ll remember”

Every Linux box I’ve owned ends up the same way. I install a package to fix something, tweak a config file in /etc, add a mount, forget about all of it, and then a year later I’m rebuilding the machine and trying to reverse engineer my own setup from shell history. If I’m lucky I wrote some of it down in Obsidian. Usually I didn’t.

That’s the thing NixOS fixes. The whole system is described in a few .nix files. If it’s not in the config, it’s not on the machine. That sounds like a small idea until you’ve rebuilt a server from scratch and had it come back exactly how it was.

Where I started: the desktop

My first real NixOS setup was my desktop, sage-system. NixOS 24.11, KDE Plasma 6, Home Manager for my user, flakes for everything. Steam and Lutris for games, PipeWire for audio, and a btrfs game drive with zstd compression mounted under my home folder. My zsh aliases lived in Nix too, which still makes me happy for some reason:

shellAliases = {
  ll = "ls -l";
  rcp = "rsync --progress --human-readable";
  cd = "z";  # zoxide
  weather = "curl wttr.in";
};

Rebuilding was one command:

sudo nixos-rebuild switch --flake .#sage-system

My desktop is on Fedora with Hyprland these days (that’s its own post), but the NixOS habit stuck. It just moved to where it makes the most sense, which is the servers.

astrid

These days NixOS runs astrid, a VM on fenrir, my Proxmox host. fenrir has an Intel Arc Pro B50 that’s split into virtual GPUs with SR-IOV, and astrid gets one of them. It does Jellyfin with hardware transcoding, a Gitea Actions runner, and mounts storage from the NAS. The GPU side has its own post: Splitting an Arc Pro B50 between VMs on Proxmox.

The B50 is a Battlemage card, and support for it in Linux is pretty new, so astrid runs linuxPackages_latest instead of the default kernel. Add intel-media-driver and vpl-gpu-rt to hardware.graphics and Jellyfin can actually use it. In NixOS that’s a few lines in one file, and if a kernel update ever breaks the card, rolling back is one reboot. The whole thing is a flake with one host and a handful of modules:

flake.nix              -> nixosConfigurations.astrid
  hosts/astrid/        -> hardware, hostname, GPU stuff
  modules/             -> boot, network, users, packages
    services/          -> jellyfin, gitea, gitea-runner, nfs

The hosts/ folder is shaped so I can add more machines later without reorganizing everything. Right now there’s one. That’s fine.

A few things I’ve landed on that I really like:

Stable by default, unstable when I need it. The flake pins nixos-25.11 but also pulls in nixos-unstable, and passes it to every module through specialArgs. So the base system stays boring, and if I need a newer version of one specific package I can grab it from pkgs-unstable without moving the whole box.

Services are tiny. This is the entire Jellyfin module:

{
  services.jellyfin = {
    enable = true;
    openFirewall = true;
  };
}

Compare that to the compose files I keep for the Docker hosts. Both are fine, but there’s something nice about the service, the firewall rule and the user all coming from one line.

The runner runs on the host. The Gitea runner on astrid uses a nixos:host label, which means jobs run directly on the machine instead of in a container. That’s on purpose. It lets workflow steps use nix, ssh and the runner’s SSH key without me building a custom image just to deploy things.

It upgrades itself. system.autoUpgrade runs every morning at 04:00 against the flake. The catch is that it’s hard coded to /etc/nixos#astrid, so the repo has to actually live there. Move it somewhere else and upgrades just quietly stop, which is a fun one to debug.

What’s annoying about it

I’m not going to pretend it’s perfect. The Nix language is weird and the error messages can be rough. Some things that are one command on Fedora turn into reading somebody’s GitHub issue from 2022 to figure out the right option name. Secrets are still something you have to think about, since anything in the Nix store is world readable. My runner token lives in a file on disk with 0600 permissions instead of in the config, and it has to exist before the service starts or the unit just fails.

And honestly, the first time you try to package something that isn’t in nixpkgs, it’s humbling.

Why it’s worth it anyway

When something breaks on astrid I don’t SSH in and start poking at it. I look at the repo. If a change made it worse, I roll back to the previous generation from the boot menu and figure it out later. Every change is a commit, so I can see exactly when I broke something and what I was thinking at the time.

That’s the same reason I like Terraform at work. The config is the documentation, and the documentation can’t drift from reality because it is the reality. NixOS is just that idea applied to the whole operating system, and once you get used to it it’s really hard to go back.