mirror of
https://git.proxmox.com/git/pve-docs
synced 2025-06-14 13:01:54 +00:00
import storage docs
This commit is contained in:
Dietmar Maurer
parent
b8a8bec810
commit
aa039b0f5a
127
pve-storage-dir.adoc
Normal file
127
pve-storage-dir.adoc
Normal file
@ -0,0 +1,127 @@
|
|||||||
|
Directory Backend
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
Storage pool type: `dir`
|
||||||
|
|
||||||
|
{pve} can use local directories or locally mounted shares for
|
||||||
|
storage. A directory is a file level storage, so you can store any
|
||||||
|
content type like virtual disk images, containers, templates, ISO images
|
||||||
|
or backup files.
|
||||||
|
|
||||||
|
NOTE: You can mount additional storages via standard linux '/etc/fstab',
|
||||||
|
and then define a directory storage for that mount point. This way you
|
||||||
|
can use any file system supported by Linux.
|
||||||
|
|
||||||
|
This backend assumes that the underlying directory is POSIX
|
||||||
|
compatible, but nothing else. This implies that you cannot create
|
||||||
|
snapshots at the storage level. But there exists a woraround for VM
|
||||||
|
images using the `qcow2` file format, because that format supports
|
||||||
|
snapshots internally.
|
||||||
|
|
||||||
|
TIP: Some storage types does not support `O_DIRECT`, so you can't use
|
||||||
|
cache mode `none` with such storages. Simply use cache mode
|
||||||
|
`writeback` instead.
|
||||||
|
|
||||||
|
We use a predefined directory layout to store different content types
|
||||||
|
into different sub-directories. This layout is use by all file level
|
||||||
|
storage backends.
|
||||||
|
|
||||||
|
.Directory layout
|
||||||
|
[width="100%",cols="d,m",options="header"]
|
||||||
|
|===========================================================
|
||||||
|
|Content type |Subdir
|
||||||
|
|VM images |images/<VMID>/
|
||||||
|
|ISO images |template/iso/
|
||||||
|
|Container templates |template/cache
|
||||||
|
|Backup files |dump/
|
||||||
|
|===========================================================
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This backend supports all common storage properties, and adds an
|
||||||
|
additional property called `path` to specify the directory. This
|
||||||
|
needs to be an absolute file system path.
|
||||||
|
|
||||||
|
.Configuration Example ('/etc/pve/storage.cfg')
|
||||||
|
----
|
||||||
|
dir: backup
|
||||||
|
path /mnt/backup
|
||||||
|
content backup
|
||||||
|
maxfiles 7
|
||||||
|
----
|
||||||
|
|
||||||
|
Above configuration defines a storage pool called `backup`. That pool
|
||||||
|
can be used to store up to 7 backups (`maxfiles 7`) per VM. The real
|
||||||
|
path for the backup files is '/mnt/backup/dump/...'.
|
||||||
|
|
||||||
|
|
||||||
|
File naming conventions
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This backend uses a well defined naming scheme for VM images:
|
||||||
|
|
||||||
|
vm-<VMID>-<NAME>.<FORMAT>
|
||||||
|
|
||||||
|
`<VMID>`::
|
||||||
|
|
||||||
|
This specifies the owner VM.
|
||||||
|
|
||||||
|
`<NAME>`::
|
||||||
|
|
||||||
|
This scan be an arbitrary name (`ascii`) without white spaces. The
|
||||||
|
backend uses `disk[N]` as default, where `[N]` is replaced by an
|
||||||
|
integer to make the name unique.
|
||||||
|
|
||||||
|
`<FORMAT>`::
|
||||||
|
|
||||||
|
Species the image format (`raw|qcow2|vmdk`).
|
||||||
|
|
||||||
|
When you create a VM template, all VM images are renamed to indicate
|
||||||
|
that they are now read-only, and can be uses as base image for clones:
|
||||||
|
|
||||||
|
base-<VMID>-<NAME>.<FORMAT>
|
||||||
|
|
||||||
|
NOTE: Such base images are used to generate cloned images. So it is
|
||||||
|
important that those files are read-only, and never gets modified. The
|
||||||
|
backend changes access mode to `0444`, and sets the immutable flag
|
||||||
|
(`chattr +i`) if the storage supports that.
|
||||||
|
|
||||||
|
Storage Features
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
As mentioned above, most file systems does not support snapshots out
|
||||||
|
of the box. To workaround that problem, this backend is able to use
|
||||||
|
`qcow2` internal snapshot capabilities.
|
||||||
|
|
||||||
|
Same applies to clones. The backend uses the `qcow2` base image
|
||||||
|
feature to create clones.
|
||||||
|
|
||||||
|
.Storage features for backend `dir`
|
||||||
|
[width="100%",cols="m,m,3*d",options="header"]
|
||||||
|
|==============================================================================
|
||||||
|
|Content types |Image formats |Shared |Snapshots |Clones
|
||||||
|
|images rootdir vztempl iso backup |raw qcow2 vmdk subvol |no |qcow2 |qcow2
|
||||||
|
|==============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
Examples
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Please use the following command to allocate a 4GB image on storage `local`:
|
||||||
|
|
||||||
|
# pvesm alloc local 100 vm-100-disk10.raw 4G
|
||||||
|
Formatting '/var/lib/vz/images/100/vm-100-disk10.raw', fmt=raw size=4294967296
|
||||||
|
sucessfuly created 'local:100/vm-100-disk10.raw'
|
||||||
|
|
||||||
|
NOTE: The image name must conform to above naming conventions.
|
||||||
|
|
||||||
|
The real file system path is shown with:
|
||||||
|
|
||||||
|
# pvesm path local:100/vm-100-disk10.raw
|
||||||
|
/var/lib/vz/images/100/vm-100-disk10.raw
|
||||||
|
|
||||||
|
And you can remove the image with:
|
||||||
|
|
||||||
|
# pvesm free local:100/vm-100-disk10.raw
|
||||||
|
|
||||||
65
pve-storage-glusterfs.adoc
Normal file
65
pve-storage-glusterfs.adoc
Normal file
@ -0,0 +1,65 @@
|
|||||||
|
GlusterFS Backend
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
Storage pool type: `glusterfs`
|
||||||
|
|
||||||
|
GlusterFS is a salable network file system. The system uses a modular
|
||||||
|
design, runs on commodity hardware, and can provide a highly available
|
||||||
|
enterprise storage at low costs. Such system is capable of scaling to
|
||||||
|
several petabytes, and can handle thousands of clients.
|
||||||
|
|
||||||
|
NOTE: After a node/brick crash, GlusterFS does a full 'rsync' to make
|
||||||
|
sure data is consistent. This can take a very long time with large
|
||||||
|
files, so this backend is not suitable to store large VM images.
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The backend supports all common storage properties, and adds the
|
||||||
|
following GlusterFS specific options:
|
||||||
|
|
||||||
|
`server`::
|
||||||
|
|
||||||
|
GlusterFS volfile server IP or DNS name.
|
||||||
|
|
||||||
|
`server2`::
|
||||||
|
|
||||||
|
Backup volfile server IP or DNS name.
|
||||||
|
|
||||||
|
`volume`::
|
||||||
|
|
||||||
|
GlusterFS Volume.
|
||||||
|
|
||||||
|
`transport`::
|
||||||
|
|
||||||
|
GlusterFS transport: `tcp`, `unix` or `rdma`
|
||||||
|
|
||||||
|
|
||||||
|
.Configuration Example ('/etc/pve/storage.cfg')
|
||||||
|
----
|
||||||
|
glusterfs: Gluster
|
||||||
|
server 10.2.3.4
|
||||||
|
server2 10.2.3.5
|
||||||
|
volume glustervol
|
||||||
|
content images,iso
|
||||||
|
----
|
||||||
|
|
||||||
|
File naming conventions
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The directory layout and the file naming conventions are inhertited
|
||||||
|
from the `dir` backend.
|
||||||
|
|
||||||
|
Storage Features
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The storage provides a file level interface, but no native
|
||||||
|
snapshot/clone implementation.
|
||||||
|
|
||||||
|
.Storage features for backend `glusterfs`
|
||||||
|
[width="100%",cols="m,m,3*d",options="header"]
|
||||||
|
|==============================================================================
|
||||||
|
|Content types |Image formats |Shared |Snapshots |Clones
|
||||||
|
|images vztempl iso backup |raw qcow2 vmdk |yes |qcow2 |qcow2
|
||||||
|
|==============================================================================
|
||||||
|
|
||||||
81
pve-storage-iscsi.adoc
Normal file
81
pve-storage-iscsi.adoc
Normal file
@ -0,0 +1,81 @@
|
|||||||
|
http://www.open-iscsi.org/[Open-iSCSI] initiator
|
||||||
|
------------------------------------------------
|
||||||
|
|
||||||
|
Storage pool type: `iscsi`
|
||||||
|
|
||||||
|
iSCSI is a widely employed technology used to connect to storage
|
||||||
|
servers. Almost all storage vendors support iSCSI. There are also open
|
||||||
|
source iSCSI target solutions available,
|
||||||
|
e.g. http://www.openmediavault.org/[OpenMediaVault], which is based on
|
||||||
|
Debian.
|
||||||
|
|
||||||
|
To use this backend, you need to install the 'open-iscsi'
|
||||||
|
package. This is a standard Debian package, but it is not installed by
|
||||||
|
default to save resources.
|
||||||
|
|
||||||
|
# apt-get install open-iscsi
|
||||||
|
|
||||||
|
Low-level iscsi management task can be done using the 'iscsiadm' tool.
|
||||||
|
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The backend supports the common storage properties `content`, `nodes`,
|
||||||
|
`disable`, and the following iSCSI specific properties:
|
||||||
|
|
||||||
|
portal::
|
||||||
|
|
||||||
|
iSCSI portal (IP or DNS name with optional port).
|
||||||
|
|
||||||
|
target::
|
||||||
|
|
||||||
|
iSCSI target.
|
||||||
|
|
||||||
|
|
||||||
|
.Configuration Example ('/etc/pve/storage.cfg')
|
||||||
|
----
|
||||||
|
iscsi: mynas
|
||||||
|
portal 10.10.10.1
|
||||||
|
target iqn.2006-01.openfiler.com:tsn.dcb5aaaddd
|
||||||
|
content none
|
||||||
|
----
|
||||||
|
|
||||||
|
TIP: If you want to use LVM on top of iSCSI, it make sense to set
|
||||||
|
`content none`. That way it is not possible to create VMs using iSCSI
|
||||||
|
LUNs directly.
|
||||||
|
|
||||||
|
File naming conventions
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The iSCSI protocol does not define an interface to allocate or delete
|
||||||
|
data. Instead, that needs to be done on the target side and is vendor
|
||||||
|
specific. The target simply exports them as numbered LUNs. So {pve}
|
||||||
|
iSCSI volume names just encodes some information about the LUN as seen
|
||||||
|
by the linux kernel.
|
||||||
|
|
||||||
|
|
||||||
|
Storage Features
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
iSCSI is a block level type storage, and provides no management
|
||||||
|
interface. So it is usually best to export one big LUN, and setup LVM
|
||||||
|
on top of that LUN. You can then use the LVM plugin to manage the
|
||||||
|
storage on that iSCSI LUN.
|
||||||
|
|
||||||
|
.Storage features for backend `iscsi`
|
||||||
|
[width="100%",cols="m,m,3*d",options="header"]
|
||||||
|
|==============================================================================
|
||||||
|
|Content types |Image formats |Shared |Snapshots |Clones
|
||||||
|
|images none |raw |yes |no |no
|
||||||
|
|==============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
Examples
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Scan a remote iSCSI portal, and returns a list of possible targets:
|
||||||
|
|
||||||
|
pvesm iscsiscan -portal <HOST[:PORT]>
|
||||||
|
|
||||||
|
|
||||||
40
pve-storage-iscsidirect.adoc
Normal file
40
pve-storage-iscsidirect.adoc
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
User Mode iSCSI Backend
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
Storage pool type: `iscsidirect`
|
||||||
|
|
||||||
|
This backend provides basically the same functionality as the
|
||||||
|
Open-iSCSI backed, but uses a user-level library (package 'libiscsi2')
|
||||||
|
to implement it.
|
||||||
|
|
||||||
|
It should be noted that there are no kernel drivers involved, so this
|
||||||
|
can be viewed as performance optimization. But this comes with the
|
||||||
|
drawback that you cannot use LVM on top of such iSCSI LUN. So you need
|
||||||
|
to manage all space allocations at the storage server side.
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The user mode iSCSI backend uses the same configuration options as the
|
||||||
|
Open-iSCSI backed.
|
||||||
|
|
||||||
|
.Configuration Example ('/etc/pve/storage.cfg')
|
||||||
|
----
|
||||||
|
iscsidirect: faststore
|
||||||
|
portal 10.10.10.1
|
||||||
|
target iqn.2006-01.openfiler.com:tsn.dcb5aaaddd
|
||||||
|
----
|
||||||
|
|
||||||
|
Storage Features
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
NOTE: This backend works with VMs only. Containers cannot use this
|
||||||
|
driver.
|
||||||
|
|
||||||
|
.Storage features for backend `iscsidirect`
|
||||||
|
[width="100%",cols="m,m,3*d",options="header"]
|
||||||
|
|==============================================================================
|
||||||
|
|Content types |Image formats |Shared |Snapshots |Clones
|
||||||
|
|images |raw |yes |no |no
|
||||||
|
|==============================================================================
|
||||||
|
|
||||||
87
pve-storage-lvm.adoc
Normal file
87
pve-storage-lvm.adoc
Normal file
@ -0,0 +1,87 @@
|
|||||||
|
LVM Backend
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Storage pool type: `lvm`
|
||||||
|
|
||||||
|
LVM is a thin software layer on top of hard disks and partitions. It
|
||||||
|
can be used to split available disk space into smaller logical
|
||||||
|
volumes. LVM is widely used on Linux and makes managing hard drives
|
||||||
|
easier.
|
||||||
|
|
||||||
|
Another use case is to put LVM on top of a big iSCSI LUN. That way you
|
||||||
|
can easily manage space on that iSCSI LUN, which would not be possible
|
||||||
|
otherwise, because the iSCSI specification does not define a
|
||||||
|
management interface for space allocation.
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The LVM backend supports the common storage properties `content`, `nodes`,
|
||||||
|
`disable`, and the following LVM specific properties:
|
||||||
|
|
||||||
|
`vgname`::
|
||||||
|
|
||||||
|
LVM volume group name. This must point to an existing volume group.
|
||||||
|
|
||||||
|
`base`::
|
||||||
|
|
||||||
|
Base volume. This volume is automatically activated before accessing
|
||||||
|
the storage. This is mostly useful when the LVM volume group resides
|
||||||
|
on a remote iSCSI server.
|
||||||
|
|
||||||
|
`saferemove`::
|
||||||
|
|
||||||
|
Zero-out data when removing LVs. When removing a volume, this makes
|
||||||
|
sure that all data gets erased.
|
||||||
|
|
||||||
|
`saferemove_throughput`::
|
||||||
|
|
||||||
|
Wipe throughput ('cstream -t' parameter value).
|
||||||
|
|
||||||
|
.Configuration Example ('/etc/pve/storage.cfg')
|
||||||
|
----
|
||||||
|
lvm: myspace
|
||||||
|
vgname myspace
|
||||||
|
content rootdir,images
|
||||||
|
----
|
||||||
|
|
||||||
|
File naming conventions
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The backend use basically the same naming conventions as the ZFS pool
|
||||||
|
backend.
|
||||||
|
|
||||||
|
vm-<VMID>-<NAME> // normal VM images
|
||||||
|
|
||||||
|
Storage Features
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
LVM is a typical block storage, but this backend does not support
|
||||||
|
snapshot and clones. Unfortunately, normal LVM snapshots are quite
|
||||||
|
inefficient, because they interfere all writes on the whole volume
|
||||||
|
group during snapshot time.
|
||||||
|
|
||||||
|
One big advantage is that you can use it on top of a shared storage,
|
||||||
|
for example an iSCSI LUN. The backend itself implement proper cluster
|
||||||
|
wide locking.
|
||||||
|
|
||||||
|
TIP: The newer LVM-thin backend allows snapshot and clones, but does
|
||||||
|
not support shared storage.
|
||||||
|
|
||||||
|
|
||||||
|
.Storage features for backend `lvm`
|
||||||
|
[width="100%",cols="m,m,3*d",options="header"]
|
||||||
|
|==============================================================================
|
||||||
|
|Content types |Image formats |Shared |Snapshots |Clones
|
||||||
|
|images rootdir |raw |possible |no |no
|
||||||
|
|==============================================================================
|
||||||
|
|
||||||
|
Examples
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
List available volume groups:
|
||||||
|
|
||||||
|
# pvesm lvmscan
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
76
pve-storage-nfs.adoc
Normal file
76
pve-storage-nfs.adoc
Normal file
@ -0,0 +1,76 @@
|
|||||||
|
NFS Backend
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Storage pool type: `nfs`
|
||||||
|
|
||||||
|
The NFS backend is based on the directory backend, so it shares most
|
||||||
|
properties. The directory layout and the file naming conventions are
|
||||||
|
the same. The main advantage is that you can directly configure the
|
||||||
|
NFS server properties, so the backend can mount the share
|
||||||
|
automatically. There is no need to modify '/etc/fstab'. The backend
|
||||||
|
can also test if the server is online, and provides a method to query
|
||||||
|
the server for exported shares.
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The backend supports all common storage properties, except the shared
|
||||||
|
flag, which is always set. Additionally, the following properties are
|
||||||
|
used to configure the NFS server:
|
||||||
|
|
||||||
|
server::
|
||||||
|
|
||||||
|
Server IP or DNS name. To avoid DNS lookup delays, it is usually
|
||||||
|
preferrable to use an IP address instead of a DNS name - unless you
|
||||||
|
have a very reliable DNS server, or list the server in the local
|
||||||
|
`/etc/hosts` file.
|
||||||
|
|
||||||
|
export::
|
||||||
|
|
||||||
|
NFS export path (as listed by `pvesm nfsscan`).
|
||||||
|
|
||||||
|
You can also set NFS mount options:
|
||||||
|
|
||||||
|
path::
|
||||||
|
|
||||||
|
The local mount point (defaults to '/mnt/pve/`<STORAGE_ID>`/').
|
||||||
|
|
||||||
|
options::
|
||||||
|
|
||||||
|
NFS mount options (see `man nfs`).
|
||||||
|
|
||||||
|
.Configuration Example ('/etc/pve/storage.cfg')
|
||||||
|
----
|
||||||
|
nfs: iso-templates
|
||||||
|
path /mnt/pve/iso-templates
|
||||||
|
server 10.0.0.10
|
||||||
|
export /space/iso-templates
|
||||||
|
options vers=3,soft
|
||||||
|
content iso,vztmpl
|
||||||
|
----
|
||||||
|
|
||||||
|
TIP: After an NFS request times out, NFS request are retried
|
||||||
|
indefinitely by default. This can lead to unexpected hangs on the
|
||||||
|
client side. For read-only content, it is worth to consider the NFS
|
||||||
|
`soft` option, which limits the number of retries to three.
|
||||||
|
|
||||||
|
|
||||||
|
Storage Features
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
NFS does not support snapshots, but the backend use `qcow2` features
|
||||||
|
to implement snapshots and cloning.
|
||||||
|
|
||||||
|
.Storage features for backend `nfs`
|
||||||
|
[width="100%",cols="m,m,3*d",options="header"]
|
||||||
|
|==============================================================================
|
||||||
|
|Content types |Image formats |Shared |Snapshots |Clones
|
||||||
|
|images rootdir vztempl iso backup |raw qcow2 vmdk subvol |yes |qcow2 |qcow2
|
||||||
|
|==============================================================================
|
||||||
|
|
||||||
|
Examples
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
You can get a list of exported NFS shares with:
|
||||||
|
|
||||||
|
# pvesm nfsscan <server>
|
||||||
88
pve-storage-rbd.adoc
Normal file
88
pve-storage-rbd.adoc
Normal file
@ -0,0 +1,88 @@
|
|||||||
|
Ceph RADOS Block Devices (RBD)
|
||||||
|
------------------------------
|
||||||
|
|
||||||
|
Storage pool type: `rbd`
|
||||||
|
|
||||||
|
http://ceph.com[Ceph] is a distributed object store and file system
|
||||||
|
designed to provide excellent performance, reliability and
|
||||||
|
scalability. RADOS block devices implement a feature rich block level
|
||||||
|
storage, and you get the following advantages:
|
||||||
|
|
||||||
|
* thin provisioning
|
||||||
|
* resizable volumes
|
||||||
|
* distributed and redundant (striped over multiple OSDs)
|
||||||
|
* full snapshot and clone capabilities
|
||||||
|
* self healing
|
||||||
|
* no single point of failure
|
||||||
|
* scalable to the exabyte level
|
||||||
|
* kernel and unser space implementation available
|
||||||
|
|
||||||
|
NOTE: For smaller deployments, it is also possible to run Ceph
|
||||||
|
services directly on your {pve} nodes. Recent hardware has plenty
|
||||||
|
of CPU power and RAM, so running storage services and VMs on same node
|
||||||
|
is possible.
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This backend supports the common storage properties `nodes`,
|
||||||
|
`disable`, `content`, and the following `rbd` specific properties:
|
||||||
|
|
||||||
|
monhost::
|
||||||
|
|
||||||
|
List of monitor daemon IPs.
|
||||||
|
|
||||||
|
pool::
|
||||||
|
|
||||||
|
Ceph pool name.
|
||||||
|
|
||||||
|
username::
|
||||||
|
|
||||||
|
RBD user Id.
|
||||||
|
|
||||||
|
krbd::
|
||||||
|
|
||||||
|
Access rbd through krbd kernel module. This is required if you want to
|
||||||
|
use the storage for containers.
|
||||||
|
|
||||||
|
.Configuration Example ('/etc/pve/storage.cfg')
|
||||||
|
----
|
||||||
|
rbd: ceph3
|
||||||
|
monhost 10.1.1.20 10.1.1.21 10.1.1.22
|
||||||
|
pool ceph3
|
||||||
|
content images
|
||||||
|
username admin
|
||||||
|
----
|
||||||
|
|
||||||
|
TIP: You can use the 'rbd' utility to do low-level management tasks.
|
||||||
|
|
||||||
|
Authentication
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
If you use cephx authentication, you need to copy the keyfile from
|
||||||
|
Ceph to Proxmox VE host.
|
||||||
|
|
||||||
|
Create the directory '/etc/pve/priv/ceph' with
|
||||||
|
|
||||||
|
mkdir /etc/pve/priv/ceph
|
||||||
|
|
||||||
|
Then copy the keyring
|
||||||
|
|
||||||
|
scp <cephserver>:/etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/<STORAGE_ID>.keyring
|
||||||
|
|
||||||
|
The keyring must be named to match your `<STORAGE_ID>`. Copying the
|
||||||
|
keyring generally requires root privileges.
|
||||||
|
|
||||||
|
Storage Features
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The `rbd` backend is a block level storage, and implements full
|
||||||
|
snapshot and clone functionality.
|
||||||
|
|
||||||
|
.Storage features for backend `rbd`
|
||||||
|
[width="100%",cols="m,m,3*d",options="header"]
|
||||||
|
|==============================================================================
|
||||||
|
|Content types |Image formats |Shared |Snapshots |Clones
|
||||||
|
|images rootdir |raw |yes |yes |yes
|
||||||
|
|==============================================================================
|
||||||
|
|
||||||
86
pve-storage-zfspool.adoc
Normal file
86
pve-storage-zfspool.adoc
Normal file
@ -0,0 +1,86 @@
|
|||||||
|
Local ZFS Pool Backend
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
Storage pool type: `zfspool`
|
||||||
|
|
||||||
|
This backend allows you to access local ZFS pools (or ZFS filesystems
|
||||||
|
inside such pools).
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The backend supports the common storage properties `content`, `nodes`,
|
||||||
|
`disable`, and the following ZFS specific properties:
|
||||||
|
|
||||||
|
pool::
|
||||||
|
|
||||||
|
Select the ZFS pool/filesystem. All allocations are done within that
|
||||||
|
pool.
|
||||||
|
|
||||||
|
blocksize::
|
||||||
|
|
||||||
|
Set ZFS blocksize parameter.
|
||||||
|
|
||||||
|
sparse::
|
||||||
|
|
||||||
|
Use ZFS thin-provisioning. A sparse volume is a volume whose
|
||||||
|
reservation is not equal to the volume size.
|
||||||
|
|
||||||
|
.Configuration Example ('/etc/pve/storage.cfg')
|
||||||
|
----
|
||||||
|
zfspool: vmdata
|
||||||
|
pool tank/vmdata
|
||||||
|
content rootdir,images
|
||||||
|
sparse
|
||||||
|
----
|
||||||
|
|
||||||
|
File naming conventions
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The backend uses the following naming scheme for VM images:
|
||||||
|
|
||||||
|
vm-<VMID>-<NAME> // normal VM images
|
||||||
|
base-<VMID>-<NAME> // template VM image (read-only)
|
||||||
|
subvol-<VMID>-<NAME> // subvolumes (ZFS filesystem for containers)
|
||||||
|
|
||||||
|
`<VMID>`::
|
||||||
|
|
||||||
|
This specifies the owner VM.
|
||||||
|
|
||||||
|
`<NAME>`::
|
||||||
|
|
||||||
|
This scan be an arbitrary name (`ascii`) without white spaces. The
|
||||||
|
backend uses `disk[N]` as default, where `[N]` is replaced by an
|
||||||
|
integer to make the name unique.
|
||||||
|
|
||||||
|
|
||||||
|
Storage Features
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
ZFS is probably the most advanced storage type regarding snapshot and
|
||||||
|
cloning. The backend uses ZFS datasets for both VM images (format
|
||||||
|
`raw`) and container data (format `subvol`). ZFS properties are
|
||||||
|
inherited from the parent dataset, so you can simply set defaults
|
||||||
|
on the parent dataset.
|
||||||
|
|
||||||
|
.Storage features for backend `zfs`
|
||||||
|
[width="100%",cols="m,m,3*d",options="header"]
|
||||||
|
|==============================================================================
|
||||||
|
|Content types |Image formats |Shared |Snapshots |Clones
|
||||||
|
|images rootdir |raw subvol |no |yes |yes
|
||||||
|
|==============================================================================
|
||||||
|
|
||||||
|
Examples
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
It is recommended to create and extra ZFS filesystem to store your VM images:
|
||||||
|
|
||||||
|
# zfs create tank/vmdata
|
||||||
|
|
||||||
|
To enable compression on that newly allocated filesystem:
|
||||||
|
|
||||||
|
# zfs set compression=on tank/vmdata
|
||||||
|
|
||||||
|
You can get a list of available ZFS filesystems with:
|
||||||
|
|
||||||
|
# pvesm zfsscan
|
||||||
328
pvesm.adoc
Normal file
328
pvesm.adoc
Normal file
@ -0,0 +1,328 @@
|
|||||||
|
include::attributes.txt[]
|
||||||
|
[[chapter-storage]]
|
||||||
|
ifdef::manvolnum[]
|
||||||
|
PVE({manvolnum})
|
||||||
|
================
|
||||||
|
|
||||||
|
NAME
|
||||||
|
----
|
||||||
|
|
||||||
|
pvesm - Proxmox VE Storage Manager
|
||||||
|
|
||||||
|
|
||||||
|
SYNOPSYS
|
||||||
|
--------
|
||||||
|
|
||||||
|
include::pvesm.1-synopsis.adoc[]
|
||||||
|
|
||||||
|
DESCRIPTION
|
||||||
|
-----------
|
||||||
|
endif::manvolnum[]
|
||||||
|
|
||||||
|
ifndef::manvolnum[]
|
||||||
|
{pve} Storage
|
||||||
|
=============
|
||||||
|
endif::manvolnum[]
|
||||||
|
|
||||||
|
The {pve} storage model is very flexible. Virtual machine images
|
||||||
|
can either be stored on one or several local storages, or on shared
|
||||||
|
storage like NFS or iSCSI (NAS, SAN). There are no limits, and you may
|
||||||
|
configure as many storage pools as you like. You can use all
|
||||||
|
storage technologies available for Debian Linux.
|
||||||
|
|
||||||
|
One major benefit of storing VMs on shared storage is the ability to
|
||||||
|
live-migrate running machines without any downtime, as all nodes in
|
||||||
|
the cluster have direct access to VM disk images. There is no need to
|
||||||
|
copy VM image data, so live migration is very fast in that case.
|
||||||
|
|
||||||
|
The storage library (package 'libpve-storage-perl') uses a flexible
|
||||||
|
plugin system to provide a common interface to all storage types. This
|
||||||
|
can be easily adopted to include further storage types in future.
|
||||||
|
|
||||||
|
|
||||||
|
Storage Types
|
||||||
|
-------------
|
||||||
|
|
||||||
|
There are basically two different classes of storage types:
|
||||||
|
|
||||||
|
Block level storage::
|
||||||
|
|
||||||
|
Allows to store large 'raw' images. It is usually not possible to store
|
||||||
|
other files (ISO, backups, ..) on such storage types. Most modern
|
||||||
|
block level storage implementations support snapshots and clones.
|
||||||
|
RADOS, Sheepdog and DRBD are distributed systems, replicating storage
|
||||||
|
data to different nodes.
|
||||||
|
|
||||||
|
File level storage::
|
||||||
|
|
||||||
|
They allow access to a full featured (POSIX) file system. They are
|
||||||
|
more flexible, and allows you to store any content type. ZFS is
|
||||||
|
probably the most advanced system, and it has full support for
|
||||||
|
snapshots and clones.
|
||||||
|
|
||||||
|
|
||||||
|
.Available storage types
|
||||||
|
[width="100%",cols="<d,1*m,4*d",options="header"]
|
||||||
|
|===========================================================
|
||||||
|
|Description |PVE type |Level |Shared|Snapshots|Stable
|
||||||
|
|ZFS (local) |zfspool |file |no |yes |yes
|
||||||
|
|Directory |dir |file |no |no |yes
|
||||||
|
|NFS |nfs |file |yes |no |yes
|
||||||
|
|GlusterFS |glusterfs |file |yes |no |yes
|
||||||
|
|LVM |lvm |block |no |no |yes
|
||||||
|
|LVM-thin |lvmthin |block |no |yes |beta
|
||||||
|
|iSCSI/kernel |iscsi |block |yes |no |yes
|
||||||
|
|iSCSI/libiscsi |iscsidirect |block |yes |no |yes
|
||||||
|
|Ceph/RBD |rbd |block |yes |yes |yes
|
||||||
|
|Sheepdog |sheepdog |block |yes |yes |beta
|
||||||
|
|DRBD9 |drbd |block |yes |yes |beta
|
||||||
|
|ZFS over iSCSI |zfs |block |yes |yes |yes
|
||||||
|
|=========================================================
|
||||||
|
|
||||||
|
TIP: It is possible to use LVM on top of an iSCSI storage. That way
|
||||||
|
you get a 'shared' LVM storage.
|
||||||
|
|
||||||
|
Storage Configuration
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
All {pve} related storage configuration is stored within a single text
|
||||||
|
file at '/etc/pve/storage.cfg'. As this file is within '/etc/pve/', it
|
||||||
|
gets automatically distributed to all cluster nodes. So all nodes
|
||||||
|
share the same storage configuration.
|
||||||
|
|
||||||
|
Sharing storage configuration make perfect sense for shared storage,
|
||||||
|
because the same 'shared' storage is accessible from all nodes. But is
|
||||||
|
also useful for local storage types. In this case such local storage
|
||||||
|
is available on all nodes, but it is physically different and can have
|
||||||
|
totally different content.
|
||||||
|
|
||||||
|
Storage Pools
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Each storage pool has a `<type>`, and is uniquely identified by its `<STORAGE_ID>`. A pool configuration looks like this:
|
||||||
|
|
||||||
|
----
|
||||||
|
<type>: <STORAGE_ID>
|
||||||
|
<property> <value>
|
||||||
|
<property> <value>
|
||||||
|
...
|
||||||
|
----
|
||||||
|
|
||||||
|
NOTE: There is one special local storage pool named `local`. It refers to
|
||||||
|
directory '/var/lib/vz' and is automatically generated at installation
|
||||||
|
time.
|
||||||
|
|
||||||
|
The `<type>: <STORAGE_ID>` line starts the pool definition, which is then
|
||||||
|
followed by a list of properties. Most properties have values, but some of them comes
|
||||||
|
with reasonable default. In that case you can omit the value.
|
||||||
|
|
||||||
|
.Default storage configuration ('/etc/pve/storage.cfg')
|
||||||
|
====
|
||||||
|
dir: local
|
||||||
|
path /var/lib/vz
|
||||||
|
content backup,iso,vztmpl,images,rootdir
|
||||||
|
maxfiles 3
|
||||||
|
====
|
||||||
|
|
||||||
|
Common Storage Properties
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
A few storage properties are common among differenty storage types.
|
||||||
|
|
||||||
|
nodes::
|
||||||
|
|
||||||
|
List of cluster node names where this storage is
|
||||||
|
usable/accessible. One can use this property to restrict storage
|
||||||
|
access to a limited set of nodes.
|
||||||
|
|
||||||
|
content::
|
||||||
|
|
||||||
|
A storage can support several content types, for example virtual disk
|
||||||
|
images, cdrom iso images, container templates or container root
|
||||||
|
directories. Not all storage types supports all content types. One can set
|
||||||
|
this property to select for what this storage is used for.
|
||||||
|
|
||||||
|
images:::
|
||||||
|
|
||||||
|
KVM-Qemu VM images.
|
||||||
|
|
||||||
|
rootdir:::
|
||||||
|
|
||||||
|
Allow to store Container data.
|
||||||
|
|
||||||
|
vztmpl:::
|
||||||
|
|
||||||
|
Container templates.
|
||||||
|
|
||||||
|
backup:::
|
||||||
|
|
||||||
|
Backup files ('vzdump').
|
||||||
|
|
||||||
|
iso:::
|
||||||
|
|
||||||
|
ISO images
|
||||||
|
|
||||||
|
shared::
|
||||||
|
|
||||||
|
Mark storage as shared.
|
||||||
|
|
||||||
|
disable::
|
||||||
|
|
||||||
|
You can use this flag to disable the storage completely.
|
||||||
|
|
||||||
|
maxfiles::
|
||||||
|
|
||||||
|
Maximal number of backup files per VM. Use `0` for unlimted.
|
||||||
|
|
||||||
|
format::
|
||||||
|
|
||||||
|
Default image format (`raw|qcow2|vmdk`)
|
||||||
|
|
||||||
|
|
||||||
|
WARNING: It is not advisable to use the same storage pool on different
|
||||||
|
{pve} clusters. Some storage operation needs exclusive access to the
|
||||||
|
storage, so proper locking is required. While this is implemented
|
||||||
|
within an cluster, it does not work between different clusters.
|
||||||
|
|
||||||
|
|
||||||
|
Volumes
|
||||||
|
-------
|
||||||
|
|
||||||
|
We use a special notation to address storage data. When you allocate
|
||||||
|
data from a storage pool, it returns such volume identifier. A volume
|
||||||
|
is identified by the `<STORAGE_ID>`, followed by a storage type
|
||||||
|
dependent volume name, separated by colon. A valid `<VOLUME_ID>` looks
|
||||||
|
like:
|
||||||
|
|
||||||
|
local:230/example-image.raw
|
||||||
|
|
||||||
|
local:iso/debian-501-amd64-netinst.iso
|
||||||
|
|
||||||
|
local:vztmpl/debian-5.0-joomla_1.5.9-1_i386.tar.gz
|
||||||
|
|
||||||
|
iscsi-storage:0.0.2.scsi-14f504e46494c4500494b5042546d2d646744372d31616d61
|
||||||
|
|
||||||
|
To get the filesystem path for a `<VOLUME_ID>` use:
|
||||||
|
|
||||||
|
pvesm path <VOLUME_ID>
|
||||||
|
|
||||||
|
Volume Ownership
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
There exists an ownership relation for 'image' type volumes. Each such
|
||||||
|
volume is owned by a VM or Container. For example volume
|
||||||
|
`local:230/example-image.raw` is owned by VM 230. Most storage
|
||||||
|
backends encodes this ownership information into the volume name.
|
||||||
|
|
||||||
|
When you remove a VM or Container, the system also remove all
|
||||||
|
associated volumes which are owned by that VM or Container.
|
||||||
|
|
||||||
|
|
||||||
|
Using the Command Line Interface
|
||||||
|
--------------------------------
|
||||||
|
|
||||||
|
I think it is required to understand the concept behind storage pools
|
||||||
|
and volume identifier, but in real life, you are not forced to do any
|
||||||
|
of those low level operations on the command line. Normally,
|
||||||
|
allocation and removal of volumes is done by the VM and Container
|
||||||
|
management tools.
|
||||||
|
|
||||||
|
Nevertheless, there is a command line tool called 'pvesm' ({pve}
|
||||||
|
storage manager), which is able to perform common storage management
|
||||||
|
tasks.
|
||||||
|
|
||||||
|
|
||||||
|
Examples
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Add storage pools
|
||||||
|
|
||||||
|
pvesm add <TYPE> <STORAGE_ID> <OPTIONS>
|
||||||
|
pvesm add dir <STORAGE_ID> --path <PATH>
|
||||||
|
pvesm add nfs <STORAGE_ID> --path <PATH> --server <SERVER> --export <EXPORT>
|
||||||
|
pvesm add lvm <STORAGE_ID> --vgname <VGNAME>
|
||||||
|
pvesm add iscsi <STORAGE_ID> --portal <HOST[:PORT]> --target <TARGET>
|
||||||
|
|
||||||
|
Disable storage pools
|
||||||
|
|
||||||
|
pvesm set <STORAGE_ID> --disable 1
|
||||||
|
|
||||||
|
Enable storage pools
|
||||||
|
|
||||||
|
pvesm set <STORAGE_ID> --disable 0
|
||||||
|
|
||||||
|
Change/set storage options
|
||||||
|
|
||||||
|
pvesm set <STORAGE_ID> <OPTIONS>
|
||||||
|
pvesm set <STORAGE_ID> --shared 1
|
||||||
|
pvesm set local --format qcow2
|
||||||
|
pvesm set <STORAGE_ID> --content iso
|
||||||
|
|
||||||
|
Remove storage pools. This does not delete any data, and does not
|
||||||
|
disconnect or unmount anything. It just removes the storage
|
||||||
|
configuration.
|
||||||
|
|
||||||
|
pvesm remove <STORAGE_ID>
|
||||||
|
|
||||||
|
Allocate volumes
|
||||||
|
|
||||||
|
pvesm alloc <STORAGE_ID> <VMID> <name> <size> [--format <raw|qcow2>]
|
||||||
|
|
||||||
|
Allocate a 4G volume in local storage. The name is auto-generated if
|
||||||
|
you pass an empty string as `<name>`
|
||||||
|
|
||||||
|
pvesm alloc local <VMID> '' 4G
|
||||||
|
|
||||||
|
Free volumes
|
||||||
|
|
||||||
|
pvesm free <VOLUME_ID>
|
||||||
|
|
||||||
|
WARNING: This really destroys all volume data.
|
||||||
|
|
||||||
|
List storage status
|
||||||
|
|
||||||
|
pvesm status
|
||||||
|
|
||||||
|
List storage contents
|
||||||
|
|
||||||
|
pvesm list <STORAGE_ID> [--vmid <VMID>]
|
||||||
|
|
||||||
|
List volumes allocated by VMID
|
||||||
|
|
||||||
|
pvesm list <STORAGE_ID> --vmid <VMID>
|
||||||
|
|
||||||
|
List iso images
|
||||||
|
|
||||||
|
pvesm list <STORAGE_ID> --iso
|
||||||
|
|
||||||
|
List container templates
|
||||||
|
|
||||||
|
pvesm list <STORAGE_ID> --vztmpl
|
||||||
|
|
||||||
|
Show filesystem path for a volume
|
||||||
|
|
||||||
|
pvesm path <VOLUME_ID>
|
||||||
|
|
||||||
|
// backend documentation
|
||||||
|
|
||||||
|
include::pve-storage-dir.adoc[]
|
||||||
|
|
||||||
|
include::pve-storage-nfs.adoc[]
|
||||||
|
|
||||||
|
include::pve-storage-glusterfs.adoc[]
|
||||||
|
|
||||||
|
include::pve-storage-zfspool.adoc[]
|
||||||
|
|
||||||
|
include::pve-storage-lvm.adoc[]
|
||||||
|
|
||||||
|
include::pve-storage-iscsi.adoc[]
|
||||||
|
|
||||||
|
include::pve-storage-iscsidirect.adoc[]
|
||||||
|
|
||||||
|
include::pve-storage-rbd.adoc[]
|
||||||
|
|
||||||
|
|
||||||
|
ifdef::manvolnum[]
|
||||||
|
include::pve-copyright.adoc[]
|
||||||
|
endif::manvolnum[]
|
||||||
|
|
||||||
Loading…
Reference in New Issue
Block a user