This repository includes all of the configuration and documentation for my home lab cluster.

This cluster consists of 4 Intel Nucs running Talos Linux with a Synology NAS for large data. More information about this repository and cluster is available at https://infrastructure.btkostner.io.

Features

• Talos Linux cluster with NVMe as a boot drive and SSD for data
• Argo CD autopilot for cluster bootstrapping
• Cilium as a kube proxy replacement and sidecar-less networking
• Rook Ceph for stateful replicated storage for all nodes
• Velero for offsite cluster backup

Media services

• Dispatcharr runs in the media namespace at https://dispatcharr.btkostner.network through the internal gateway. Its all-in-one container stores application data, PostgreSQL, and recordings on a 10Gi ceph-safe-block volume mounted at /data. External traffic and DNS from the pod are routed through Mullvad WireGuard by a Gluetun native sidecar with its firewall kill switch enabled; Dispatcharr starts only after Gluetun is healthy. HTTP ingress on port 9191 and in-cluster pod/service traffic (10.244.0.0/16, 10.96.0.0/12) are allowed outside the tunnel. To refresh the gluetun-credentials item in 1Password, generate a Mullvad WireGuard configuration for an active account and device, then copy Interface.PrivateKey into private-key and Interface.Address into addresses in that same configuration. If Interface.Address contains multiple CIDR values, enter them comma-separated. The dispatcharr-vpn ExternalSecret maps these fields to Gluetun's WIREGUARD_PRIVATE_KEY and WIREGUARD_ADDRESSES environment variables. Create the first admin account in the web interface, then add M3U and EPG sources. In-cluster clients can use http://dispatcharr.media.svc.cluster.local. Argo CD discovers cluster/media/dispatcharr/kustomization.yaml after it is pushed to main.

Provisioning

This page goes over how to setup a whole new cluster. Starting with basic networking setup, Talos configuration generating and applying, and Argo CD bootstrapping.

My home cluster is made up of 4x Intel 11th gen i5 NUCs with 32 GB of ram, an NVMe boot drive, and a 1TB SSD for Ceph. All of the nodes are nearly identical to make things easier, though this isn't a hard requirement. The main reason I chose this configuration is for it's low power usage, small space, and Intel Xe graphics.

I also have a PiKVM setup connected to a TESmart 4K 16 Ports HDMI KVM Switch. This allows easy bios control and booting of images. This makes it infinitely easier to install Talos on a cluster.

Node Setup

Node setup is mostly ensuring all of the hardware is working and has up to date firmware. This is especially important for my Crucial MX500 SSDs as older versions can randomly disconnect and cause Ceph to go into an unhealthy state.

Once all of the hardware is confirmed working, I set a static IP for the node. This starts at 192.168.3.11 and continue for each node. Talos will set a virtual IP at 192.168.3.10 for the control plane nodes.

One important thing to note is I use cluster.btkostner.network to point to my cluster virtual IP of 192.168.3.10. Yes this is a local address so it will only resolve correctly in my network, but that should be fine.

Generating Talos Configuration

Talos configuration resides in the provision/talos/ directory. It includes a folder of configuration patches as well as a generate.sh script to run the needed talosctl commands.

If you plan to run this for your own cluster, ensure all of the patches in provision/talos/patches are relevant to you.

Once you are ready, you can run the generate.sh script. This will generate your controlplane.yaml, talosconfig, and worker.yaml files. Ensure you back these files.

Installing Talos

For installing Talos, I grab the latest Talos ISO from GitHub, upload it to my PiKVM, and boot the node into it. This pulls up the Talos instance in bootstrap status. I then run this command for the node:

talosctl apply --file ./controlplane.yaml --nodes "192.168.3.11" --insecure

This sets up the control node. Then run this command to bootstrap the cluster:

talosctl bootstrap --nodes "192.168.3.11"

This does the initial Kubernetes resource creation and what not. At this point, 192.168.3.10 should point to the control plane node.

At this point, I can run the same talosctl apply command for all of the other control nodes. Then I apply the worker configuration with:

talosctl apply --file ./worker.yaml --nodes "192.168.3.14" --insecure

With all of that done, there is now a fresh Kubernetes cluster. Note that because the default CNI is not installed, none of the node networking will work and every Kubernetes node will have a taint on it. They will also reboot every 10 minutes until the Cilium CNI is installed.

Finally I run this command to generate the needed kubectl entry:

talosctl kubeconfig ~/.kube/config --nodes talos.btkostner.network --force

Installing Core Resources

Once the Kubernetes cluster is up, we can start installing software on it. To simplify this process I just run the provision/core/install.sh script. This will install all of the resources in the cluster/argocd directory. Once this is ran initially, ArgoCD will take care of syncing all resources, so you should never need to run this provision core install script again.

Bootstrapping 1Password credentials

The last step of provisioning a cluster is setting up 1Password connect to handle secrets in Kubernetes. Luckily Argo will install the 1Password connect service and the external secrets operator, but we have to add the required 1Password secret for your vault. To do this, I have an install script at provision/1password/install.sh. This will use the 1Password CLI to verify if the 1Password Connect password exists in the 1Password vault (or create it if it doesn't), then copies the password to Kubernetes. If you are running it on your own, please verify the variables in the script before running. It makes heavy assumptions based on my own 1Password setup.

Networking

This page covers networking for individual nodes, our private internal DNS and gateway, and our public DNS and gateways.

Local Network

My Intel NUC cluster runs on a separate vlan powered by Ubiquiti.

Private Network

Private services live under btkostner.network and are reached with Cloudflare Zero Trust. Devices run the Cloudflare One (WARP) client. A cloudflared deployment in the cluster (cluster/networking/cloudflared) connects the cluster tunnel to Cloudflare.

The tunnel has a private hostname route for btkostner.network, which also covers its subdomains. When a WARP client looks up a name such as grafana.btkostner.network, Cloudflare Gateway answers with a temporary IP in 172.64.128.0/20 and sends the traffic through the tunnel. The public btkostner.network zone has no A, AAAA, or CNAME records.

cloudflared resolves the real address through Blocky (192.168.3.61). Blocky's customDNS mapping in cluster/networking/blocky answers btkostner.network and every subdomain with the internal-gateway IP (192.168.3.52). More specific entries override this for hosts that are not behind the gateway: talos (the Kubernetes API on the control plane nodes), nuc-1 to nuc-5, kvm, and behemoth. Add an entry there when a new host needs to be reachable.

Gateway TCP and UDP proxying must stay enabled in Zero Trust for private hostname routing to work. The default device profile reconnects WARP 10 minutes after it is turned off (auto_connect: 600).

Tautulli must have HTTP Proxy enabled (http_proxy=1 in its persistent /config/config.ini) so redirects use HTTPS behind the gateway.

Talos must be able to start without Blocky or the Cloudflare tunnel. The shared configuration in provision/talos/talconfig.yaml uses Cloudflare's NTP IPs (162.159.200.1 and 162.159.200.123) and local host entries that map talos.btkostner.network to the three control-plane addresses (192.168.3.11–192.168.3.13). Keep these settings independent of cluster DNS. After a full power loss, start the network equipment, all three control-plane nodes, and both workers. Blocky must recover before cloudflared can resolve its destinations.

Personal MCP portal

The shared endpoint is https://mcp.btkostner.io/mcp. Cloudflare MCP server portals combine the servers behind one authenticated endpoint. Clients do not need WARP. Private upstreams remain private.

The Kubernetes server is defined in cluster/developer/kubernetes-mcp. It has its own ServiceAccount and serves Streamable HTTP through the internal gateway. OpenCode's manifests use the same server through http://kubernetes-mcp.developer.svc.cluster.local:8080/mcp, without the public portal or a local MCP subprocess. Its Home Assistant connection is unchanged.

The Argo CD server is defined in cluster/developer/argocd-mcp. It runs native HTTP transport from argocd-mcp@0.9.0 with MCP_READ_ONLY=true. Its dedicated Argo CD account has API-key capability only and is bound to role:readonly. The account and RBAC configuration must reach main and sync before issuing a token.

The static Argo CD token is stored only in the Cloudflare upstream server's encrypted credentials, not in Git, 1Password, or a Kubernetes workload Secret. Configure authentication type Bearer with the JSON-encoded custom-header form {"headers":{"x-argocd-api-token":"<token>"}}. Cloudflare supplies this header on upstream requests; the MCP pod has no token environment variable. The token has no automatic expiry. Rotate it by issuing a replacement, updating Cloudflare's encrypted credentials, and revoking the old account token. Cloudflare's API does not permit changing a server's authentication type in place; choose Bearer when creating this registration.

The Cloudflare server uses its own upstream OAuth connection. In Zero Trust → Access controls → MCP Portals → MCP servers, connect or re-authenticate Cloudflare and approve the required account permissions. This one-time administrator approval is separate from login to the personal portal. Until approval completes, the server remains waiting and its tools are unavailable. A local MCP client's existing OAuth session does not transfer to the portal.

For the initial rollout, deploy the new Kubernetes MCP app before applying the OpenCode migration. Argo CD discovers the new app directory after the manifests reach main.

Cloudflare configuration is managed through the Cloudflare API or dashboard, not by Argo CD:

• Account: d6d3937679a6366e10e8914df0a191b2; portal ID: personal.
• DNS: proxied CNAME mcp.btkostner.io → gateway.agents.cloudflare.com. Do not put the portal under the private btkostner.network route.
• Both the portal and each server have an Access application. The reusable Personal MCP access policy permits only btkostner@gmail.com, through the existing Cloudflare identity provider.
• Managed OAuth is enabled. Access tokens last 15 minutes; refresh grants last 14 days. Clients must support OAuth token refresh to avoid repeated login prompts.
• Dynamic client registration accepts localhost and loopback callbacks, plus HTTPS callbacks on claude.ai, chatgpt.com, chat.openai.com, platform.openai.com, and playground.ai.cloudflare.com. Add the callback URI required by another client to the portal Access application's Managed OAuth settings.
• Gateway routing is enabled on the portal and each private upstream. Kubernetes, Argo CD, and Home Assistant use private hostnames routed through the internal Gateway. No 192.168.2.30/32 CIDR route is required.
• Each portal-server mapping has Require user auth off (on_behalf: false). Servers use their shared upstream identity or administrator OAuth credential, not a separate upstream login for each client.

The Home Assistant MCP URL has a secret path. Its full value remains in 1Password and Cloudflare's server configuration. Do not copy it into documentation, client configuration, or logs. Cloudflare labels this upstream unauthenticated because it does not use an Authorization header; possession of its private URL still grants access.

Connect a client

For clients with remote MCP and OAuth support, add the portal URL and sign in with the permitted Cloudflare account:

{
  "mcpServers": {
    "personal": {
      "url": "https://mcp.btkostner.io/mcp"
    }
  }
}

The surrounding configuration format varies by client. For a local client that supports only stdio MCP, use an OAuth-capable HTTP bridge such as mcp-remote:

{
  "mcpServers": {
    "personal": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://mcp.btkostner.io/mcp"]
    }
  }
}

An LLM product must support MCP, or an MCP bridge, to use this endpoint. A public URL does not add MCP support to a product that has none. Hosted clients cannot run a local stdio bridge.

Cloudflare currently rejects the default Python urllib user agent with error 1010. Node, HTTPX, and curl reach the OAuth challenge. The endpoint does not guarantee support for every HTTP client or LLM product.

For unattended clients, create a separate Access service token for each client. Attach a Service Auth (non_identity) policy for that token to the portal Access application and every upstream server Access application it needs. Keep on_behalf: false, and send CF-Access-Client-Id and CF-Access-Client-Secret headers. Keep these values in the client's secret store. No permanent service token is created by this setup.

Add a server

1. Deploy a Streamable HTTP MCP endpoint. Private portal connections through Gateway do not support legacy SSE-only servers. A stdio-only server needs an HTTP wrapper.
2. Make private endpoints reachable through a private hostname route or a narrow CIDR route. Keep credentials in 1Password, the upstream OAuth provider, or Cloudflare's encrypted upstream credentials.
3. In Zero Trust → Access controls → MCP Portals → MCP servers, register the URL, configure its authentication, and enable Route traffic through Cloudflare Gateway. Wait for Ready.
4. Add an Access application of type mcp, with a via_mcp_server_portal destination for the server ID. Attach Personal MCP access, and any required client-specific Service Auth policies.
5. Add the server to portal personal. Select the tools and permissions intentionally. Keep Require user auth off for shared private credentials; enable it when a server needs each user's upstream OAuth identity.
6. Start a new client session, or use the portal's server-selection tools, to enable the new server. Verify a read through the portal, not only the server's sync status.

Tools are prefixed with their server ID, so names do not collide. The portal enables Code Mode by default to reduce the tool definitions sent to the model. Clients that need normal tool definitions can use https://mcp.btkostner.io/mcp?codemode=off.

Public Network

For public networking it's a pretty standard Kubernetes setup. One abnormal thing about my setup is I use the new (and totally awesome) Kubernetes Gateway API. There is a single external-gateway resource that uses MetalLB to allocate the 192.68.3.50 IP address to it. Port forwarding with Ubiquiti allows my house public IP address to accept traffic and route to my cluster services.

To make this fully work, a ddns job on the cluster keeps the abraxis.tv Cloudflare records pointed at my house public IP address. btkostner.network has no public records and is only reachable through Cloudflare Zero Trust.

Secrets

As you may have noticed, nothing in this repository is a secret. There are no encrypted files. This is because we use 1Password to manage all of our secrets in the cluster. This is done via 1Password connect and external secrets operator.

In order to get this working as intended, there is one provisioning step that needs to run in order to add the required 1Password authentication into the cluster. Once that is done, everything else should be automated and ready to go.

Storage

This cluster uses Rook Ceph to provide distributed storage across all nodes. Each Intel NUC contributes its SSD to a shared Ceph cluster, which is then exposed to workloads via Kubernetes StorageClasses. An external Synology NAS provides additional NFS storage for large media files.

Ceph Cluster

The Ceph cluster is deployed via the rook-ceph Helm chart (v1.20.9) in the rook-ceph namespace. It uses all available nodes and devices automatically.

Key features enabled:

• Dashboard — accessible via HTTP route for cluster monitoring
• Monitoring — Prometheus metrics are exported; the chart's monitoring.enabled value also generates the namespace-scoped ServiceMonitor RBAC used by the operator.
• Disk prediction — the diskprediction_local module is enabled for drive health forecasting
• PG autoscaler — automatically tunes placement group counts
• CSI read affinity — reads are served from the closest OSD based on topology
• Discovery daemon — automatically detects new devices

The Ceph cluster data directory is stored at /var/lib/rook on each node.

OSD Maintenance

Rook checks whether an OSD can stop before it updates the deployment. The single-copy danger-block-pool can block this check even when all placement groups are active and clean. For a maintenance window, temporarily increase that CephBlockPool's replication size from 1 to 2 and wait for all placement groups to become active and clean. Keep Rook's safety checks enabled. Restore size 1 only after the CephCluster is Ready, its observed generation matches its current generation, and all OSD deployments are ready.

Use a temporary Argo sync-deny window that selects only rook-ceph while the pool differs from Git. Remove that window after the original replication size is restored. Preserve retained volumes and bucket data.

Block Pools

Block pools define how data is replicated across the Ceph cluster. Two pools exist with different durability guarantees.

block-pool (Non-Replicated)

Warning: This pool has no data redundancy. If a single OSD or node is lost, data in this pool is lost. Use only for data that is easily recreatable or non-critical.

safe-block-pool (Replicated)

This pool stores three copies of every block across different hosts, surviving up to two simultaneous host failures.

Object Store

object-store-replicated (S3-Compatible)

A Ceph Object Store providing S3-compatible storage with the following configuration:

Pool preservation is enabled (preservePoolsOnDelete: true), so data pools are retained even if the object store resource is deleted.

Storage Classes

Storage classes are the primary interface for workloads to request storage. The cluster defines three storage classes.

Block Storage Classes

All block storage classes use the rook-ceph.rbd.csi.ceph.com provisioner with ext4 filesystem, Retain reclaim policy, Immediate volume binding, and volume expansion enabled.

Object Storage Classes

Choosing a Storage Class

• ceph-block — Use for non-critical, easily recreatable data where performance matters more than durability (1× replication).
• ceph-block-replicated — Use for important application data that must survive node failures (3× replication).
• ceph-object-replicated — Use for S3-compatible object/bucket storage with erasure coding.

Local Storage

The local-path storage class (rancher local-path-provisioner v0.0.37, cluster/kube-system/local-path-provisioner/) provisions hostPath volumes under /var/mnt/local-path-provisioner on nodes labeled btkostner.network/local-storage: "true" (nuc-1 through nuc-4 only, set via per-node nodeLabels in provision/talos/talconfig.yaml; nuc-5 is intentionally excluded pending a disk-layout check, since its Talos install disk and Ceph's OSD device filter both resolve to /dev/sda). The directory lives on the Talos EPHEMERAL partition (/var), the same partition Rook's dataDirHostPath uses — there is no dedicated partition or CSI-level isolation from other /var I/O. volumeBindingMode: WaitForFirstConsumer co-locates the pod and its volume; there is no resize or snapshot support at the CSI level, by design — CNPG's own replica-recreate procedure and the Barman Cloud plugin backups substitute for those.

The nodes also still carry the legacy cnpg.io/local-storage: "true" label (same 4 nodes), which immich-database's nodeSelector still targets as of this writing. The generic label was added alongside it during a staged migration; consumers should move to btkostner.network/local-storage and the legacy label should be removed from talconfig.yaml once nothing references it.

The CloudNativePG operator (cluster/cnpg-system/cnpg/) manages per-cluster Cluster resources; immich-database (cluster/media/immich/resources/database.yaml) is currently the only one and uses local-path for its storage instead of Ceph RBD, relying on PostgreSQL streaming replication (instances: 3, podAntiAffinityType: required across kubernetes.io/hostname) for availability instead of storage-layer replication. Losing a node's local volume does not strand the cluster: the operator reseeds the lost instance from a surviving primary via pg_basebackup. Backups and WAL archiving go through the Barman Cloud CNPG-I plugin (cluster/cnpg-system/plugin-barman-cloud/) to the safe-object-store Ceph RGW, using the same ObjectBucketClaim pattern as Loki and Mimir.

NFS Storage

The Synology NAS provides NFS volumes for large media and download directories. NFS PVCs use the nfs storage class with ReadWriteMany access mode, allowing multiple pods to mount the same volume simultaneously. These are typically sized at 1Mi as nominal placeholders since the actual storage is managed by the NAS. Static NFS media layout:

• /volume2/Media/downloads backs the download claims for Lidarr, Radarr, SABnzbd, and Sonarr.
• /volume2/Media backs the media claims and the Scryer and Weaver data claims.
• /volume2/homes/btkostner/Photos backs the Immich photo library claim.

Weaver stores incomplete downloads and unpacking work in a disk-backed emptyDir capped at 100Gi and scheduled only to nodes with non-rotational storage. A one-time init container moves the prior NAS intermediate workspace into the local workspace before the first cutover. Later Pod recreation or rescheduling does not retain the local workspace. Its weaver-media NFS claim remains mounted only at /download/downloads/complete.

Volume Inventory

By Namespace

developer

download

home

media

rook-ceph (via Helm)

Summary by Storage Class

Largest Consumers

Configuration Reference

All Rook Ceph resources are defined in:

cluster/rook-ceph/rook-ceph/
├── kustomization.yaml
├── values.yaml
└── resources/
    ├── ceph-block-pool.yaml
    ├── ceph-block-pool-replicated.yaml
    ├── ceph-cluster.yaml
    ├── ceph-object-store-replicated.yaml
    ├── storage-class-ceph-block.yaml
    ├── storage-class-ceph-block-replicated.yaml
    └── storage-class-ceph-object-replicated.yaml

Individual application PVCs are defined alongside their deployments in:

cluster/<namespace>/<application>/resources/pvc-*.yaml

Cilium Upgrades

Upgrading Cilium is... painful. Because it runs as the networking back bone it's very easy to do something wrong and completely break your cluster. As always, read the official Cilium upgrade documentation, but this is how I've been upgrading it so far:

1. Run the preflight checks with helm:

helm --kube-context admin@btkostner install cilium-preflight cilium/cilium --version 1.15.0 --namespace=kube-system --set preflight.enabled=true --set agent=false --set operator.enabled=false

This will pull all of the container images for each node to reduce issues.

2. Condon all of the nodes. This will prevent them from trying to upgrade in place and causing a huge cluster wide outage.

3. Apply the upgrade. This should be done via Argo and merging in a helm version upgrade. Make note to ensure the helm values file is updated for the new version.

4. You should see the cilium operator container start up on the cluster. Make sure no errors are present.

5. Start rebooting nodes. I usually start with the head controller node which is 192.168.3.11 on my cluster. You can accomplish a standard reboot with the talosctl utility like so:

talosctl reboot --debug --wait --timeout 30s -n 192.168.3.11

Note, if you run into issues, it's helpful to use the kvm console to see what's going on. You can also force a reboot with Ctrl+Alt+Del.

This should reboot the node safely and bring it back into the cluster. Because it's the same host, it will still be condoned on startup. You can uncondone it and check the status of the cilium agent on the node. It should be running and healthy.

6. Continue rebooting all of the other nodes. Same as above, though now that controller node is back up, you should not get any weird dashboard outages or issues.

7. Once all nodes are rebooted, you can uncondone them all and the cluster should be back to normal.