Storage

This cluster uses Rook Ceph to provide distributed storage across all nodes. Each Intel NUC contributes its SSD to a shared Ceph cluster, which is then exposed to workloads via Kubernetes StorageClasses. An external Synology NAS provides additional NFS storage for large media files.

Ceph Cluster

The Ceph cluster is deployed via the rook-ceph Helm chart (v1.19.3) in the rook-ceph namespace. It uses all available nodes and devices automatically.

Key features enabled:

• Dashboard — accessible via HTTP route for cluster monitoring
• Monitoring — Prometheus metrics are exported
• Disk prediction — the diskprediction_local module is enabled for drive health forecasting
• PG autoscaler — automatically tunes placement group counts
• CSI read affinity — reads are served from the closest OSD based on topology
• Discovery daemon — automatically detects new devices

The Ceph cluster data directory is stored at /var/lib/rook on each node.

Block Pools

Block pools define how data is replicated across the Ceph cluster. Two pools exist with different durability guarantees.

block-pool (Non-Replicated)

Warning: This pool has no data redundancy. If a single OSD or node is lost, data in this pool is lost. Use only for data that is easily recreatable or non-critical.

safe-block-pool (Replicated)

This pool stores three copies of every block across different hosts, surviving up to two simultaneous host failures.

Object Store

object-store-replicated (S3-Compatible)

A Ceph Object Store providing S3-compatible storage with the following configuration:

Pool preservation is enabled (preservePoolsOnDelete: true), so data pools are retained even if the object store resource is deleted.

Storage Classes

Storage classes are the primary interface for workloads to request storage. The cluster defines three storage classes.

Block Storage Classes

All block storage classes use the rook-ceph.rbd.csi.ceph.com provisioner with ext4 filesystem, Retain reclaim policy, Immediate volume binding, and volume expansion enabled.

Object Storage Classes

Choosing a Storage Class

• ceph-block — Use for non-critical, easily recreatable data where performance matters more than durability (1× replication).
• ceph-block-replicated — Use for important application data that must survive node failures (3× replication).
• ceph-object-replicated — Use for S3-compatible object/bucket storage with erasure coding.

NFS Storage

The Synology NAS provides NFS volumes for large media and download directories. NFS PVCs use the nfs storage class with ReadWriteMany access mode, allowing multiple pods to mount the same volume simultaneously. These are typically sized at 1Mi as nominal placeholders since the actual storage is managed by the NAS. Static NFS media layout:

• /volume2/Media/downloads backs the download claims for Lidarr, Radarr, SABnzbd, and Sonarr.
• /volume2/Media backs the media claims and the Scryer and Weaver data claims.

Weaver stores incomplete downloads and unpacking work in a disk-backed emptyDir capped at 100Gi and scheduled only to nodes with non-rotational storage. A one-time init container moves the prior NAS intermediate workspace into the local workspace before the first cutover. Later Pod recreation or rescheduling does not retain the local workspace. Its weaver-media NFS claim remains mounted only at /download/downloads/complete.

Volume Inventory

By Namespace

developer

download

home

media

rook-ceph (via Helm)

Summary by Storage Class

Largest Consumers

Configuration Reference

All Rook Ceph resources are defined in:

cluster/rook-ceph/rook-ceph/
├── kustomization.yaml
├── values.yaml
└── resources/
    ├── ceph-block-pool.yaml
    ├── ceph-block-pool-replicated.yaml
    ├── ceph-cluster.yaml
    ├── ceph-object-store-replicated.yaml
    ├── storage-class-ceph-block.yaml
    ├── storage-class-ceph-block-replicated.yaml
    └── storage-class-ceph-object-replicated.yaml

Individual application PVCs are defined alongside their deployments in:

cluster/<namespace>/<application>/resources/pvc-*.yaml