Need Ceph design or a review? WZ-IT provides Proxmox Ceph planning, setup and health checks. Book an initial consultation.
This guide runs from planning through installation and OSDs to the pool and acceptance, with the commands from the Proxmox documentation. Agree backups, rollback and a maintenance window before changing production clusters. As of October 2026.
Contents
- 1. Plan requirements and failure domains
- 2. Install Ceph and configure networks
- 3. Create MONs, MGRs and OSDs
- 4. Check the pool and RBD storage
- 5. Health check and acceptance
- Maintenance: OSDs and nodes
- Operations and next steps
- Sources
1. Plan requirements and failure domains
Plan at least three Proxmox nodes, suitable OSD disks and capacity reserves for failures. Ceph replicates data; the CRUSH rule must distribute copies across different hosts. A three-replica pool needs more raw capacity than usable data space. The storage comparison helps decide whether Ceph fits.
Proxmox recommends at least 10 Gbps for Ceph traffic. In particular, separate Corosync from storage traffic. Budget memory per OSD in addition to guests and other services; Proxmox recommends 8 GiB per OSD. Expose disks directly, using HBAs rather than hardware RAID for SAS/SATA. See the Proxmox Ceph documentation.
2. Install Ceph and configure networks
Open Ceph on a cluster node and start the installation wizard. Choose a Ceph version compatible with PVE and existing nodes, and select the appropriate repository. Configure the public network for client and service communication; an additional cluster network separates replication and heartbeats. Shared configuration is distributed across the cluster. Install required packages on every intended node.
For OVHcloud, plan the network over vRack separately: keep Ceph addresses and MTU consistent across nodes and do not treat VLAN separation alone as physical independence for Corosync. Check the contracted network capacity against requirements before accepting Ceph.
On the command line this corresponds to the following steps (adjust networks and addresses):
pveceph install # on every node
pveceph init --network 10.10.10.0/24 --cluster-network 10.10.20.0/24 # once on one node
--network sets the public network, --cluster-network the additional network for replication and heartbeats. Without the second parameter Ceph uses one network for both.
3. Create MONs, MGRs and OSDs
Distribute monitors across hosts and check their quorum. Create managers and then OSDs on the intended unused disks. OSD creation dedicates the disk to Ceph; verify device identity and existing data before approval. WAL/DB placement, device classes and CPU reserves belong in the design before production data is written.
Plan three MONs on three hosts; check MGR redundancy separately. A QDevice replaces neither a Ceph monitor nor a storage host. Follow the Proxmox documentation linked above for installation and service steps.
The commands:
pveceph mon create # on every monitor node, at least three
pveceph mgr create # on at least two nodes for redundancy
pveceph osd create /dev/sdb # per disk
pveceph osd create /dev/sdc -db_dev /dev/nvme0n1 # optionally with a separate DB/WAL device
4. Check the pool and RBD storage
Create a replicated pool with the intended CRUSH rule. For this three-host design, size 3/min_size 2 is the starting point. Verify the actual settings and add the pool as RBD storage for VM disks. Do not reduce replication as an improvised response to insufficient space. The Ceph pool documentation explains both parameters.
The pool can be created in the web interface or on the command line and attached as RBD storage right away:
pveceph pool create vm-pool --size 3 --min_size 2 --add_storages
pveceph pool ls
Never set min_size to 1: during failures writes could go to a single copy, risking data loss. The number of placement groups can be left to the PG autoscaler, which the Proxmox documentation recommends.
5. Health check and acceptance
These read-only commands provide inspection data, not predetermined test results:
ceph status
ceph health detail
ceph osd tree
ceph osd df
ceph df
ceph osd pool ls detail
Assess monitor quorum, OSD states, placement groups, failure domains, replication and free space together. HEALTH_OK alone demonstrates neither application performance nor recoverability. Also inspect the configured full/nearfull thresholds rather than assuming a default.
| Area | Acceptance criterion |
|---|---|
| MON and MGR | Expected MONs in quorum, active MGR and intended redundancy |
| OSDs and PGs | Intended OSDs up and in, PGs active+clean after recovery completes |
| CRUSH and pool | Copies distributed across hosts, size and min_size match the design |
| Capacity | Per-OSD free space checked with ceph osd df, reserves for planned failures checked |
The Ceph monitoring documentation explains states and commands. ceph osd df provides utilization for each OSD; ceph df shows aggregate cluster and pool values (Ceph: Free drive space). Record deviations, causes and actions rather than merely silencing warnings.
A failure test requires an approved test plan, abort criteria and sufficient spare capacity. With exactly three hosts, the third host copy can only be restored when the failed host returns or is replaced. Do not trigger production incidents for demonstration.
Maintenance: OSDs and nodes
Before maintenance on a node, the noout flag stops Ceph from immediately redistributing missing OSDs. For single OSDs, check before stopping whether that is safe:
ceph osd set noout # before maintenance
ceph osd ok-to-stop 3 # may OSD 3 be stopped?
ceph osd out 3 # take the OSD out of data distribution
ceph osd unset noout # after maintenance
Replace a failed disk by removing the OSD and recreating it on the new device with pveceph osd create. Wait until the placement groups are active+clean again before touching the next node.
Operations and next steps
Ceph design and health checks establish the design. Managed Proxmox covers the explicitly agreed ongoing Ceph scope. Plan backups and recovery separately with Proxmox Backup Server.
