mirror of
https://git.proxmox.com/git/pve-docs
synced 2025-04-28 17:49:53 +00:00
ceph: follow up rewords & fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This commit is contained in:
parent
76f6eca4ee
commit
2f19a6b0b6
43
pveceph.adoc
43
pveceph.adoc
@ -79,17 +79,16 @@ Check also the recommendations from
|
|||||||
http://docs.ceph.com/docs/luminous/start/hardware-recommendations/[Ceph's website].
|
http://docs.ceph.com/docs/luminous/start/hardware-recommendations/[Ceph's website].
|
||||||
|
|
||||||
.CPU
|
.CPU
|
||||||
As higher the core frequency the better, this will reduce latency. Among other
|
Higher CPU core frequency reduce latency and should be preferred. As a simple
|
||||||
things, this benefits the services of Ceph, as they can process data faster.
|
rule of thumb, you should assign a CPU core (or thread) to each Ceph service to
|
||||||
To simplify planning, you should assign a CPU core (or thread) to each Ceph
|
provide enough resources for stable and durable Ceph performance.
|
||||||
service to provide enough resources for stable and durable Ceph performance.
|
|
||||||
|
|
||||||
.Memory
|
.Memory
|
||||||
Especially in a hyper-converged setup, the memory consumption needs to be
|
Especially in a hyper-converged setup, the memory consumption needs to be
|
||||||
carefully monitored. In addition to the intended workload (VM / Container),
|
carefully monitored. In addition to the intended workload from virtual machines
|
||||||
Ceph needs enough memory to provide good and stable performance. As a rule of
|
and container, Ceph needs enough memory available to provide good and stable
|
||||||
thumb, for roughly 1TiB of data, 1 GiB of memory will be used by an OSD. With
|
performance. As a rule of thumb, for roughly 1 TiB of data, 1 GiB of memory
|
||||||
additionally needed memory for OSD caching.
|
will be used by an OSD. OSD caching will use additional memory.
|
||||||
|
|
||||||
.Network
|
.Network
|
||||||
We recommend a network bandwidth of at least 10 GbE or more, which is used
|
We recommend a network bandwidth of at least 10 GbE or more, which is used
|
||||||
@ -97,15 +96,14 @@ exclusively for Ceph. A meshed network setup
|
|||||||
footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
|
footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server]
|
||||||
is also an option if there are no 10 GbE switches available.
|
is also an option if there are no 10 GbE switches available.
|
||||||
|
|
||||||
To be explicit about the network, since Ceph is a distributed network storage,
|
The volume of traffic, especially during recovery, will interfere with other
|
||||||
its traffic must be put on its own physical network. The volume of traffic
|
services on the same network and may even break the {pve} cluster stack.
|
||||||
especially during recovery will interfere with other services on the same
|
|
||||||
network.
|
|
||||||
|
|
||||||
Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb
|
Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb
|
||||||
link, a SSD or a NVMe SSD certainly can. Modern NVMe SSDs will even saturate 10
|
link, multiple HDD OSDs per node can, and modern NVMe SSDs will even saturate
|
||||||
Gb of bandwidth. You also should consider higher bandwidths, as these tend to
|
10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwith
|
||||||
come with lower latency.
|
will ensure that it isn't your bottleneck and won't be anytime soon, 25, 40 or
|
||||||
|
even 100 GBps are possible.
|
||||||
|
|
||||||
.Disks
|
.Disks
|
||||||
When planning the size of your Ceph cluster, it is important to take the
|
When planning the size of your Ceph cluster, it is important to take the
|
||||||
@ -114,16 +112,21 @@ might take long. It is recommended that you use SSDs instead of HDDs in small
|
|||||||
setups to reduce recovery time, minimizing the likelihood of a subsequent
|
setups to reduce recovery time, minimizing the likelihood of a subsequent
|
||||||
failure event during recovery.
|
failure event during recovery.
|
||||||
|
|
||||||
In general SSDs will provide more IOPs then spinning disks. This fact and the
|
In general SSDs will provide more IOPs than spinning disks. This fact and the
|
||||||
higher cost may make a xref:pve_ceph_device_classes[class based] separation of
|
higher cost may make a xref:pve_ceph_device_classes[class based] separation of
|
||||||
pools appealing. Another possibility to speedup OSDs is to use a faster disk
|
pools appealing. Another possibility to speedup OSDs is to use a faster disk
|
||||||
as journal or DB/WAL device, see xref:pve_ceph_osds[creating Ceph OSDs]. If a
|
as journal or DB/WAL device, see xref:pve_ceph_osds[creating Ceph OSDs]. If a
|
||||||
faster disk is used for multiple OSDs, a proper balance between OSD and WAL /
|
faster disk is used for multiple OSDs, a proper balance between OSD and WAL /
|
||||||
DB (or journal) disk must be selected, otherwise the faster disk becomes the
|
DB (or journal) disk must be selected, otherwise the faster disk becomes the
|
||||||
bottleneck for all linked OSDs.
|
bottleneck for all linked OSDs.
|
||||||
|
|
||||||
Aside from the disk type, Ceph best performs with an even sized and distributed
|
Aside from the disk type, Ceph best performs with an even sized and distributed
|
||||||
amount of disks per node. For example, 4x disks à 500 GB in each node.
|
amount of disks per node. For example, 4 x 500 GB disks with in each node is
|
||||||
|
better than a mixed setup with a single 1 TB and three 250 GB disk.
|
||||||
|
|
||||||
|
One also need to balance OSD count and single OSD capacity. More capacity
|
||||||
|
allows to increase storage density, but it also means that a single OSD
|
||||||
|
failure forces ceph to recover more data at once.
|
||||||
|
|
||||||
.Avoid RAID
|
.Avoid RAID
|
||||||
As Ceph handles data object redundancy and multiple parallel writes to disks
|
As Ceph handles data object redundancy and multiple parallel writes to disks
|
||||||
@ -137,8 +140,8 @@ the ones from Ceph.
|
|||||||
WARNING: Avoid RAID controller, use host bus adapter (HBA) instead.
|
WARNING: Avoid RAID controller, use host bus adapter (HBA) instead.
|
||||||
|
|
||||||
NOTE: Above recommendations should be seen as a rough guidance for choosing
|
NOTE: Above recommendations should be seen as a rough guidance for choosing
|
||||||
hardware. Therefore, it is still essential to test your setup and monitor
|
hardware. Therefore, it is still essential to adapt it to your specific needs,
|
||||||
health & performance.
|
test your setup and monitor health and performance continuously.
|
||||||
|
|
||||||
|
|
||||||
[[pve_ceph_install]]
|
[[pve_ceph_install]]
|
||||||
|
Loading…
Reference in New Issue
Block a user