How many nodes does a Ceph cluster on Proxmox need?

The minimum is three nodes - which matches the requirement for Proxmox HA clusters (at least 3 nodes plus shared storage). Three nodes allow the replication default of size 3 / min_size 2: three copies of every object, with I/O continuing as long as two are available. More nodes improve fault tolerance, rebalancing behavior and capacity.

Ceph or ZFS - which is better for Proxmox?

They are different tools. Ceph is shared storage for 3+ nodes: live migration without copying VM data, self-healing, scale-out. ZFS is local storage - for 2-node setups or smaller environments combined with storage replication (pvesr): ZFS-only, asynchronous, minimum interval 1 minute, and the latest state may be lost on failover. If you want true HA with shared storage, you end up with Ceph; if you run 2 nodes, with ZFS plus replication.

What network does Ceph need?

At least 10 Gbps exclusively for Ceph traffic - sharing with VM or Corosync traffic is not an option, especially since a single modern NVMe can saturate 10 Gbps. High-performance setups separate further: 25+ Gbps for the Ceph-internal network, 10+ Gbps public and 1 Gbps dedicated to Corosync.

Can Ceph be retrofitted into an existing Proxmox cluster?

Yes, provided the hardware fits: at least 3 nodes, dedicated disks for the OSDs on HBAs (no RAID controllers), sufficient RAM (at least 8 GiB per OSD recommended) and an exclusive network with at least 10 Gbps. In practice, the network is the most common retrofit effort - which is exactly what we assess upfront during planning.

What happens when a disk fails?

Ceph heals itself: the affected OSD goes down and the data is redistributed from the remaining replicas. With size 3 / min_size 2, I/O continues in the meantime. It is important to wait until the rebalance has finished and enough replicas exist again before further intervention - for us, that is part of the monitored standard procedure.

Do you also operate Ceph clusters long-term?

Yes. Ceph can be included in the agreed Managed Proxmox operations scope. This can cover 24/7 proactive monitoring of OSD, MON, and pool health, updates in maintenance windows, fill-level and capacity planning, and the agreed incident response. Managed Proxmox starts at €179.90 per node and month; the Ceph scope is confirmed in the proposal after the technical assessment.

Proxmox Ceph: design, implementation, and operations

We design Ceph around nodes, disks, networking, capacity, and failure objectives, build the cluster, and can provide the agreed ongoing operations.

3+ nodes, no SAN
Live migration without copying
Self-healing storage

Book an intro call When Ceph fits

Reviews

IT Beratung Service Consulting Soforthilfe

Companies worldwide trust WZ-IT

Read client reviews

Decision

When Ceph - and when ZFS is enough

Ceph is not an end in itself. The honest answer depends on node count, budget and whether you actually need shared storage.

Ceph

Ceph fits when ...

You want to migrate running VMs without downtime - Ceph/RBD is shared storage, VM data does not need to be copied for this.
You are planning an HA cluster: it requires at least 3 nodes plus shared storage - Ceph delivers both in one design, without any external SAN.
Capacity and redundancy should grow with the cluster - Ceph scales horizontally via additional nodes and disks.
You want to run hyperconverged: storage lives directly on the compute nodes, no separate storage appliance needed.

ZFS

ZFS + replication fits when ...

You only have 2 nodes or a tight budget - Ceph is not built for that, ZFS with storage replication is the realistic alternative here.
ZFS is local storage, not shared storage - replication (pvesr) partially compensates for this, but works exclusively with ZFS.
Replication runs asynchronously with a minimum interval of 1 minute - the latest state may be lost on failover. Your workloads must tolerate this gap.
You do not need live migration without copying - with local ZFS, VM data is transferred when moving.

The detailed comparison including LVM: Proxmox storage: ZFS, Ceph & LVM compared

Design

Design decisions that get expensive later

A Ceph cluster forgives a lot in operations - but little in design. These six points decide stability and performance before the first byte is written.

Network

Network first
Ceph needs at least 10 Gbps - exclusively for storage traffic. High-performance setups separate further: 25+ Gbps Ceph-internal, 10+ Gbps public and 1 Gbps dedicated to Corosync. A single modern NVMe can already saturate 10 Gbps.

Hardware

HBA instead of RAID controller
The docs are unambiguous: avoid RAID controllers, use host bus adapters (HBA) instead. RAID logic interferes with Ceph's own redundancy and recovery mechanisms rather than complementing them.
RAM per OSD
Current recommendation: at least 8 GiB of RAM per OSD, the default is 4 GiB. The old 1 GB per TB rule of thumb is outdated - size by OSD count, not by capacity.
Do not forget CPU
Plan at least one core or thread per Ceph service - on top of your VMs. OSDs also benefit from high clock speeds, not just from many cores.

Data

Replication: size 3 / min_size 2
The default means: three copies of every object, I/O continues as long as two are available. Cutting down on replicas cuts exactly the redundancy Ceph is built for.

Version

Choose the Ceph version deliberately
Proxmox VE 9.2 ships Ceph Squid 19.2.3 and Tentacle 20.2.1 - new installations default to Tentacle. The version choice belongs in the planning phase, not in the aftermath.

Operations

Operations: what Ceph means day to day

Self-healing & rebalancing
If an OSD fails, Ceph redistributes the data on its own. Important in operations: wait until the rebalance has finished and enough replicas exist again before intervening further.
Fill-level discipline
The nearfull_ratio warns at 85% by default - a Ceph cluster should be planned well below that capacity-wise and monitored continuously.
Erasure coding: possible, not standard
EC yields more usable capacity - for the price of performance. RBD on EC pools additionally requires a replicated pool for metadata. An expert topic, not a default setup.
24/7 proactive monitoring
We proactively monitor OSD, MON, and pool health around the clock and address relevant anomalies before they become incidents.
Updates in maintenance windows
Ceph and PVE updates coordinated, node by node, with an eye on cluster state - never all at once, never blind.

Health check

Typical Ceph misconfigurations from our health checks

We see these five patterns again and again when auditing existing Ceph clusters - each one costs stability, performance or both.

Patterns

What we find in existing clusters

Ceph and Corosync on the same network - Corosync needs low latency; if it shares the wire with storage traffic, I/O load endangers cluster communication.
RAID controllers underneath the OSDs - exactly what the Ceph docs advise against. Use HBAs and leave redundancy to Ceph.
size 2 instead of 3 „to save space“ - undermines the redundancy Ceph is built for and turns every failure into a risk.
Cluster regularly above 85% fill level - beyond the nearfull_ratio, things get tight for recovery and rebalancing.
10 Gbps network shared with VM traffic - the docs require the Ceph network to be exclusive; a single modern NVMe can saturate the link on its own.

Existing Ceph cluster?

Unsure how your Ceph is doing?

Network
Replication
Fill levels
Hardware

We audit existing Proxmox and Ceph environments in a structured health check - network, replication, fill levels, hardware - and take over operations afterwards if desired. Even without documentation.

Proxmox takeover →

Operations

Ceph within Managed Proxmox operations

Ceph can be included in Managed Proxmox. Monitoring of OSD, MON, and pool health, updates, capacity planning, and incident response are confirmed as a specific operations scope after the technical assessment.

Managed Proxmoxfrom €179.90 excl. VAT / node and monthBase price; the Ceph scope is confirmed in the proposal after the technical assessment.

Book an intro call View Managed Proxmox

Other routes

For professional secrecy holders

Operate Proxmox in the Section 203 Managed Cloud

If Proxmox processes client, patient or other professional secrets, WZ-IT plans the application, infrastructure, access, backups and ongoing operations within a purpose-built operating model.

View the Section 203 Managed Cloud →

Frequently Asked Questions

Answers to the most important questions

Knowledge

Further Proxmox guides

In-depth knowledge from our Proxmox knowledge base.

Guides

Matching Solutions for Your Project

Reviews & projects

Proxmox proof from real projects

Customer feedback on virtualization, backup, VPN, security and ongoing cluster operations.

WZ-IT moved our studio infrastructure from decentralised individual devices to a central platform: every site is securely connected via VPN, new devices are onboarded automatically and an entire site is provisioned from a template, without manual steps on location. What impressed me most is the breadth and depth of their knowledge: Timo and Robin are not a typical IT provider who sets up a server and leaves. The two of them think their way into highly complex infrastructure and software topics, work through every requirement we put in front of them, and build networking, provisioning and operations so that everything fits together in the end. WZ-IT is an excellent partner for complex software, network and architecture projects.

Steve KirchnerManaging Director, nextGYM GmbH

View project

International

Built in Germany's Ruhr Valley. Running worldwide.

WZ-IT designs, develops and operates infrastructure and software for clients in Germany and internationally. We deliver projects remotely and continue supporting them in ongoing operations after go-live.

Selected projects

Read client reviews

Enquiry

Assess a Proxmox Ceph architecture

Name nodes, drives, networking, capacity, and required failure behaviour.

Straight with Timo and Robin - no sales team, no pitch
An honest take, including when we are not the right fit
Concrete next steps for infrastructure, software or AI

No risk: worst case, you leave with a clearer understanding of your project than before.

WZ-IT's advice on our Azure migration was technically sound and completely non-binding right from the intro call - we took away a great deal.

Jakob ÖschlbergerInno7 GmbH

Email[email protected]Phone+49 2922 875068

WZ-IT's advice on our Azure migration was technically sound and completely non-binding right from the intro call - we took away a great deal.

Jakob ÖschlbergerInno7 GmbH

Proxmox Ceph: design, implementation, and operations