Storage

This cluster uses Rook Ceph to provide distributed storage across all nodes. Each Intel NUC contributes its SSD to a shared Ceph cluster, which is then exposed to workloads via Kubernetes StorageClasses. An external Synology NAS provides additional NFS storage for large media files.

Ceph Cluster

The Ceph cluster is deployed via the rook-ceph Helm chart (v1.20.9) in the rook-ceph namespace. It uses all available nodes and devices automatically.

Key features enabled:

• Dashboard — accessible via HTTP route for cluster monitoring
• Monitoring — Prometheus metrics are exported; the chart's monitoring.enabled value also generates the namespace-scoped ServiceMonitor RBAC used by the operator.
• Disk prediction — the diskprediction_local module is enabled for drive health forecasting
• PG autoscaler — automatically tunes placement group counts
• CSI read affinity — reads are served from the closest OSD based on topology
• Discovery daemon — automatically detects new devices

The Ceph cluster data directory is stored at /var/lib/rook on each node.

OSD Maintenance

Rook checks whether an OSD can stop before it updates the deployment. The single-copy danger-block-pool can block this check even when all placement groups are active and clean. For a maintenance window, temporarily increase that CephBlockPool's replication size from 1 to 2 and wait for all placement groups to become active and clean. Keep Rook's safety checks enabled. Restore size 1 only after the CephCluster is Ready, its observed generation matches its current generation, and all OSD deployments are ready.

Use a temporary Argo sync-deny window that selects only rook-ceph while the pool differs from Git. Remove that window after the original replication size is restored. Preserve retained volumes and bucket data.

Block Pools

Block pools define how data is replicated across the Ceph cluster. Two pools exist with different durability guarantees.

block-pool (Non-Replicated)

Warning: This pool has no data redundancy. If a single OSD or node is lost, data in this pool is lost. Use only for data that is easily recreatable or non-critical.

safe-block-pool (Replicated)

This pool stores three copies of every block across different hosts, surviving up to two simultaneous host failures.

Object Store

object-store-replicated (S3-Compatible)

A Ceph Object Store providing S3-compatible storage with the following configuration:

Pool preservation is enabled (preservePoolsOnDelete: true), so data pools are retained even if the object store resource is deleted.

Storage Classes

Storage classes are the primary interface for workloads to request storage. The cluster defines three storage classes.

Block Storage Classes

All block storage classes use the rook-ceph.rbd.csi.ceph.com provisioner with ext4 filesystem, Retain reclaim policy, Immediate volume binding, and volume expansion enabled.

Object Storage Classes

Choosing a Storage Class

• ceph-block — Use for non-critical, easily recreatable data where performance matters more than durability (1× replication).
• ceph-block-replicated — Use for important application data that must survive node failures (3× replication).
• ceph-object-replicated — Use for S3-compatible object/bucket storage with erasure coding.

Local Storage

The local-path storage class (rancher local-path-provisioner v0.0.37, cluster/kube-system/local-path-provisioner/) provisions hostPath volumes under /var/mnt/local-path-provisioner on nodes labeled btkostner.network/local-storage: "true" (nuc-1 through nuc-4 only, set via per-node nodeLabels in provision/talos/talconfig.yaml; nuc-5 is intentionally excluded pending a disk-layout check, since its Talos install disk and Ceph's OSD device filter both resolve to /dev/sda). The directory lives on the Talos EPHEMERAL partition (/var), the same partition Rook's dataDirHostPath uses — there is no dedicated partition or CSI-level isolation from other /var I/O. volumeBindingMode: WaitForFirstConsumer co-locates the pod and its volume; there is no resize or snapshot support at the CSI level, by design — CNPG's own replica-recreate procedure and the Barman Cloud plugin backups substitute for those.

The nodes also still carry the legacy cnpg.io/local-storage: "true" label (same 4 nodes), which immich-database's nodeSelector still targets as of this writing. The generic label was added alongside it during a staged migration; consumers should move to btkostner.network/local-storage and the legacy label should be removed from talconfig.yaml once nothing references it.

The CloudNativePG operator (cluster/cnpg-system/cnpg/) manages per-cluster Cluster resources; immich-database (cluster/media/immich/resources/database.yaml) is currently the only one and uses local-path for its storage instead of Ceph RBD, relying on PostgreSQL streaming replication (instances: 3, podAntiAffinityType: required across kubernetes.io/hostname) for availability instead of storage-layer replication. Losing a node's local volume does not strand the cluster: the operator reseeds the lost instance from a surviving primary via pg_basebackup. Backups and WAL archiving go through the Barman Cloud CNPG-I plugin (cluster/cnpg-system/plugin-barman-cloud/) to the safe-object-store Ceph RGW, using the same ObjectBucketClaim pattern as Loki and Mimir.

NFS Storage

The Synology NAS provides NFS volumes for large media and download directories. NFS PVCs use the nfs storage class with ReadWriteMany access mode, allowing multiple pods to mount the same volume simultaneously. These are typically sized at 1Mi as nominal placeholders since the actual storage is managed by the NAS. Static NFS media layout:

• /volume2/Media/downloads backs the download claims for Lidarr, Radarr, SABnzbd, and Sonarr.
• /volume2/Media backs the media claims and the Scryer and Weaver data claims.
• /volume2/homes/btkostner/Photos backs the Immich photo library claim.

Weaver stores incomplete downloads and unpacking work in a disk-backed emptyDir capped at 100Gi and scheduled only to nodes with non-rotational storage. A one-time init container moves the prior NAS intermediate workspace into the local workspace before the first cutover. Later Pod recreation or rescheduling does not retain the local workspace. Its weaver-media NFS claim remains mounted only at /download/downloads/complete.

Volume Inventory

By Namespace

developer

download

home

media

rook-ceph (via Helm)

Summary by Storage Class

Largest Consumers

Configuration Reference

All Rook Ceph resources are defined in:

cluster/rook-ceph/rook-ceph/
├── kustomization.yaml
├── values.yaml
└── resources/
    ├── ceph-block-pool.yaml
    ├── ceph-block-pool-replicated.yaml
    ├── ceph-cluster.yaml
    ├── ceph-object-store-replicated.yaml
    ├── storage-class-ceph-block.yaml
    ├── storage-class-ceph-block-replicated.yaml
    └── storage-class-ceph-object-replicated.yaml

Individual application PVCs are defined alongside their deployments in:

cluster/<namespace>/<application>/resources/pvc-*.yaml