Commit Graph

1147 Commits

Author SHA1 Message Date
Tim Marx
32f749b840 storage zfs: changed return value description & optionals
Signed-off-by: Tim Marx <t.marx@proxmox.com>
2018-11-09 14:49:43 +01:00
Tim Marx
a49fc735e5 close #1949: storage zfs: changed zpool command parser
Signed-off-by: Tim Marx <t.marx@proxmox.com>
2018-11-09 14:49:43 +01:00
Tim Marx
b005f2f483 Fix: api zfs: changed return value name to errors
Signed-off-by: Tim Marx <t.marx@proxmox.com>
2018-10-29 10:33:39 +01:00
Thomas Lamprecht
a298e55665 remove outdated unused API2::Storage::Replication module
this was not installed, had references to non-existing modules
(e.g., PVE::ReplicationTools) and the things it probably was intended
for are done in pve-manager, which has full access to all pve perl
libs and API methods (from a dependency POV)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-10-03 16:58:57 +02:00
Thomas Lamprecht
0370861cfd diskmanage: rename check_unused to assert_disk_unused
to have a clear method name for this. check_XYZ suggests also that we
return true if the check was OK, but we don't.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-10-03 14:51:38 +02:00
Thomas Lamprecht
9280153e10 rename check_available to assert_sid_unused
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-10-03 14:49:14 +02:00
Dominik Csapak
4dcb16c0dc fix #1929: only check storage if user want to create one
this is useful if a user wants to create similar storage on each host

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-10-03 14:30:22 +02:00
Dominik Csapak
820bab50b9 add missing storage check in LVM Disk API
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-10-03 14:30:22 +02:00
Dominik Csapak
76c1e57be7 refactor disk/storage checks for Disk API
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-10-03 14:30:22 +02:00
Dietmar Maurer
94836d7b8c pvesm iscsiscan: fix cli parameters
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2018-10-01 10:56:18 +02:00
Thomas Lamprecht
dd9e97ed14 find_free_diskname: fixup regex match operator
Co-developed-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-19 11:21:37 +02:00
Dietmar Maurer
0057171085 Fix #1925: untaint rbd JSON output
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Reviewed-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2018-09-19 11:21:37 +02:00
Dominik Csapak
401b56fc54 fix #1912: find the partition instead of guessing it
instead of using '$dev1' as partition,
get the information from /sys/block/$dev

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-09-14 11:24:42 +02:00
Thomas Lamprecht
dd1fa860d0 get_vm_disk_number: follow up cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-14 09:19:08 +02:00
Stoiko Ivanov
f4cc2c4afe Fix #1913: get_vm_disk_number: clone plugindata to avoid side effects
Accessing a non-existing 'format' key in plugindata (e.g., in LvmThinPlugin),
created it by autovivication, thus breaking the fallback to the default value
'raw' upon any following access.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-09-14 09:17:40 +02:00
Alwin Antreich
e5b2206f8a rbd: krbd_feature_disable was not disabling features
$features is actually an array reference, so use it as one.
This broke creation and migration of disks on rbd storages

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-12 14:56:36 +02:00
Thomas Lamprecht
83a40b8d3b refactor is_valid_vm_diskname to private simpler sub
This was newly introduced and is only used once, so having a
wantarray return mechanism, without ever using it or knowing for sure
if this may help with reuse of the method is not ideal.

Make the sub a module private one just returning the vm disk number
or explicit undef. Pass it the $suffix variable, to avoid recomputing
it every time called by out caller's loop.

If there's re-use potential in the future we can actually decide what
makes sense to return.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-11 07:52:43 +02:00
Stoiko Ivanov
c4a29df483 refactor finding next diskname for all plugins
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-09-10 12:21:10 +02:00
Thomas Lamprecht
59fa9fd6a3 next diskname: start ids with 0 to honor MAX_VOLUMES_PER_GUEST
else we can only have MAX_VOLUMES_PER_GUEST-1 disk per VMID,
not tragic but possible confusing

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-10 11:21:12 +02:00
Stoiko Ivanov
c0535aa72f make max number of disks a constant
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-09-07 15:28:15 +02:00
Stoiko Ivanov
345f898108 add vm_diskname helpers (get_next, is_valid)
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-09-07 15:28:15 +02:00
Alwin Antreich
aa14def420 Addition to fix #1895, skip image if no owner
Non conforming image names are not ignored anymore by the new rbd_ls
implementation, this patch adds the old behaviour.

This fix is a temporary workaround and should be removed, once the new
image name parser is ready.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-09-07 13:51:18 +02:00
Thomas Lamprecht
1be93fe2fd rdb: followup cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-09-06 15:19:16 +02:00
Dominik Csapak
c093e93b28 rbd: remove unused size conversion function
since the json output gives the sizes in bytes, we do not
need to convert anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-09-06 15:18:23 +02:00
Dominik Csapak
89a8800bc3 fix #1895: use json for 'rbd ls -l' and 'rbd info'
since ceph changed the plain output format for 12.2.8
we have to change the code anyway, and when were at it,
we can change it to the (hopefully) more robust json output

Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-09-06 15:11:26 +02:00
Dietmar Maurer
1e17594e74 add missing parameter descriptions 2018-08-09 09:52:44 +02:00
Dietmar Maurer
48dccd2fbe cleanup return property descriptions 2018-08-09 09:29:10 +02:00
Dietmar Maurer
2654942813 PVE::API2::Storage::Content::index - document return values 2018-08-09 08:53:24 +02:00
Dominik Csapak
4d12dbffc4 add return description for zfs detail api call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-08 12:01:54 +02:00
Dominik Csapak
e39e8ee213 refactor diskmanagement lock_file calls
so that we only have one place where we reference the lockfile
and the timeout

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-08 12:01:02 +02:00
Dominik Csapak
7058abe29e add 'single' raidlevel for zfs
the syntax for creating a pool with a single disk is
not the same as for mirror, so let the user select it
explicitely

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-08 12:00:18 +02:00
Dominik Csapak
38572a8f56 rename raidlvl to raidlevel
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-08 11:56:35 +02:00
Dominik Csapak
fdc863c705 fix descriptions of api calls
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-08 11:56:17 +02:00
Dietmar Maurer
7d597888a4 cleanup descriptions 2018-08-08 08:22:27 +02:00
Dietmar Maurer
5be1a092d6 fix schema - 'string-list' is a format, not a type 2018-08-08 08:21:06 +02:00
Dominik Csapak
c84106edc9 add API for ZFS management
a list, a detail and a create api call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-08 08:02:14 +02:00
Dominik Csapak
5e35281da2 rename lvmcreate to lvmthincreate for lvm thin
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-08 07:44:14 +02:00
Dietmar Maurer
d347322a63 PVE::API2::Storage::Status::index - document return schema properties
And add property "used_fraction" - I want to use that for the CLI tool.
2018-08-07 13:24:27 +02:00
Stoiko Ivanov
ca552c7639 Fix #1858: lvm_find_free_diskname check for base
lvm_find_free_diskname only checked for existing volumes starting with 'vm-',
and not with 'base-'.

Unify implementation with other Plugins.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-08-07 12:14:11 +02:00
Thomas Lamprecht
f15ac9b5a8 LIO: followup: various small cleanups
move two loop bodies from

if (condition) {
    ...
}

too
next if !condition;

...

to save an indentation level

rename variables to a bit shorter version, i.e.:
s/oneTarget/target/
s/oneTpg/tpg/

and a comment rewording

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-08-02 14:57:07 +02:00
Thomas Lamprecht
ff69c66022 LIO: followup: shorter stderr/out logging
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-08-02 14:56:20 +02:00
Thomas Lamprecht
ccdf8ddbda LIO: followup: fix indentation 2018-08-02 14:56:20 +02:00
Thomas Lamprecht
d9254744a8 LIO: followup: remove trailing whitespaces
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-08-02 14:56:20 +02:00
Udo Rader
46c6107eb1 Linux LIO/targetcli support
Introducing LIO/targetcli support allowing to use recent linux
distributions as iSCSI targets for ZFS volumes.

In order for this to work, two preconditions have to be met:

1. the portal has to be set up correctly using targetcli
2. the initiator has to be authorized to connect to the target
   based on the initiator's InitiatorName

When adding a LIO iSCSI target, a new "LIO target portal group" field needs
to be correctly populated in the "Add: ZFS over iSCSI" popup, containing the
fitting "LIO target portal group" name (typically something like 'tpg1').

Signed-Off-By: Udo Rader <udo.rader@bestsolution.at>
Tested-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2018-08-02 14:56:20 +02:00
Dietmar Maurer
fd87487b3a fix typo 2018-08-02 12:44:30 +02:00
Dominik Csapak
793d720cd2 add API for add Directory storage
creates/lists systemd mount units for /mnt/pve/.*
filetypes allowed are ext4 and xfs for now
mount with /dev/disk/by-uuid

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 12:28:27 +02:00
Dietmar Maurer
fc1880566c cleanup: do not convert exceptions to strings. 2018-08-02 11:39:35 +02:00
Dominik Csapak
0ea9f3844e add API for LVMThin management
like the LVM API, but return an array for the list,
because we do not have nested data here

and create a vg and thin lv with the name given and use the full size

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 11:29:47 +02:00
Dominik Csapak
2c2fd98b87 add metadata_size and _used to lv list
so that we can show it in the webinterface and the user can check
how full it is

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 11:16:05 +02:00
Dominik Csapak
49d149e638 extend list_thinpools for multiple vgs and more information
if no vg is given, give back all thinpools from all vgs
if verbose is 1, then give back the information about the thinpools
(like size and free)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 11:10:33 +02:00
Dominik Csapak
83bbd6f5a1 add type and skipsmart to /nodes/NODE/disks/list
so that we can use it for a generic disk selector
this mirrors the functionality we have in
/nodes/NODE/ceph/disks api call (which we can deprecate then)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 10:53:30 +02:00
Dominik Csapak
8b6842caa2 add API for LVM management
currently only list and create,
the list is in a format so that we can use it in an extjs tree

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 10:38:27 +02:00
Dominik Csapak
8cccb3447b add an option to include pvs in lvm_vgs
this will be used for the lvm part of the disk management

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-08-02 09:38:27 +02:00
Thomas Lamprecht
074b2cb4fa remove unused Data::Dumper usages
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-11 12:23:44 +02:00
Thomas Lamprecht
5527c824dd cephtools: simplify ceph_check_keyfile 2018-07-04 16:56:24 +02:00
Thomas Lamprecht
5402cea50d cephfs plugin: followup with some code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-04 15:07:33 +02:00
Alwin Antreich
fe0d606f5e Use keyfile create/remove from CephTools
in the RBDPlugin, that is also shared by the CephFSPlugin

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:18:32 +02:00
Alwin Antreich
e34ce14443 Cephfs storage plugin
- ability to mount through kernel and fuse client
 - allow mount options
 - get MONs from ceph config if not in storage.cfg
 - allow the use of ceph config with fuse client
 - Delete secret on cephfs storage creation

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:18:19 +02:00
Alwin Antreich
3e47917203 Add simple keyring check for cephfs/rbd
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:13:21 +02:00
Alwin Antreich
9b7ba1db42 Merge RBD and CephFS code into a helper module
Some methods for connecting to a ceph cluster are the same for RBD and
CephFS, these are merged into the helper modules.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-07-04 13:13:21 +02:00
Thomas Lamprecht
6d3b36e1b0 api add: remove now uneeded $cred_file variable
was used for cleanup, which is now handled by on_delete_hook

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
Thomas Lamprecht
f9602323c6 lvm: use add hook
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
Thomas Lamprecht
2e109b4bc2 rbd: use add/delete hooks
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
Thomas Lamprecht
ab5e32bb1d cifs: use add/delete hooks
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
Thomas Lamprecht
3932ca0d1b add on_add and on_delete hooks
on_add_hook allows to encapsulate storage specific add steps, like
copying a keyring (RDB) or creating a volume group (LVM) in a clean
manner.
The same for deletion with on_delete_hook, here all should be cleaned
up, as much as possible.

Until now, this was done directly in the api config CREATE and DELETE
code, respectively, with a series of

if ($storage_type eq 'foo) {
    ...
} elsif ($storage_type eq 'bar') {
    ...
}

which isn't really that nice...

Another nice result of this approach is that also external plugins
can use those hooks and to their setup/cleanup steps sanely.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-07-03 12:27:34 +02:00
Thomas Lamprecht
d7b707626a storage add: always extract password from parameters
as else we write it to /etc/pve/storage.cfg which is readable by
www-data, a not really private group...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-06-29 13:16:17 +02:00
Thomas Lamprecht
2d4404fc63 add cifs: fix variable declaration in post if
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-06-29 13:15:26 +02:00
Dominik Csapak
42f2c57dfa replace read_password with param_mapping
we only need this for cifs as this is the only type
of storage where we expect a password

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-06-27 13:23:21 +02:00
Dietmar Maurer
8b3d5c1f12 PVE/API2/Storage/Config.pm - set type in returns schema 2018-06-22 07:52:51 +02:00
Wolfgang Link
3160dbf139 CIFS: add mkdir option allowing to switch off path creation
see commit: c7616abcb2
2018-06-14 07:49:43 +02:00
Thomas Lamprecht
5bc3edb243 followup: fix typo 2018-06-06 15:38:22 +02:00
Wolfgang Link
c2f12dc648 Use enums for smbvers to prevent incorrect input 2018-06-06 15:36:25 +02:00
Wolfgang Link
ff6fa67fb6 Make check_connection less costly.
This command will only check the needed share
and do not query the hole server shares.

This reduce the answer time and also has the benefit we check the
credentials on this share and not on the server.
2018-05-22 13:18:12 +02:00
Wolfgang Link
9a75947b49 fix #1770: allow ex/import linked clones.
We encode the base-volume-name in the volname what is not needed.
On ex/import we need the correct dataset name.
2018-05-22 11:36:36 +02:00
Wolfgang Link
840e3797a5 cifs: fix uninitialized warning in check_connection
otherwise this spams the log when check_connection runs into a timeout.
2018-05-16 10:06:00 +02:00
Wolfgang Bumiller
d68c7bca46 whitespace fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-05-14 08:35:22 +02:00
Wolfgang Link
fee2cb61e3 Fix typo. 2018-05-14 08:10:19 +02:00
Wolfgang Link
ee8c176d3a Simplify image/template upload check.
All storage's witch are able to store images must have a path.
2018-05-14 08:10:19 +02:00
Fabian Grünbichler
5fc02afb7d fix #1754: rbd: fix handling of external clusters
with the recent refactoring, external clusters were not handled
correctly with librd if a pveceph or storage specific ceph config
exists.

change the behaviour to include the pveceph config file only for pveceph
managed clusters, and a storage specific one only for external ones.

set mon_host correctly using the values from storage.cfg for external
librbd clusters.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2018-05-08 10:09:58 +02:00
Thomas Lamprecht
fc05c9a0d8 fixup error message typo: s/sould/should/
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-05-08 09:34:37 +02:00
Wolfgang Link
cf8ded6a96 fix #1691: increase timeout in worker
A ZFS storage under heavy load can take more time.
2018-05-07 12:01:20 +02:00
Wolfgang Link
894b9ecc1e fix #1691: replace udev check
`zfs create` add the creation job in a worker queue,
which should normally execute instantly. But there are circumstances
where the job will take a while to get processed.
If this is the case udev settle will see no dev in the queue and the program
will continue without an allocated dev.

The busy waiting is not best practice but the only way to be sure,
that the block device exists.
2018-05-07 12:01:20 +02:00
Alwin Antreich
0b45fb3eb5 Fix #1750: set monhost split to old behavior
The path method of the RBDPlugin got a list with comma separated monhosts,
but it needs the list with semi-colon for qemu.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-05-04 14:18:04 +02:00
Alwin Antreich
6cc88e8ef1 Refactor of method build_cmd and path
Method build_cmd and path use similar code to generate the ceph command
line or qemu config parameters. They now use the private method
ceph_connect_option for parameter generation.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-04-18 10:45:06 +02:00
Alwin Antreich
41aacc6cde Fix #1542: show storage utilization per pool
- get storage utilization per pool
 - use librados2-perl for pool status
 - add librados2-perl as build-depends and depends in debian/control

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-04-18 10:43:14 +02:00
Dietmar Maurer
566639d518 cifs: use correct pluging class name 2018-03-16 13:23:47 +01:00
Dietmar Maurer
a9db2ca8a5 cifs: move plugin related code into plugin 2018-03-16 13:08:32 +01:00
Dietmar Maurer
304344ce34 add missing "use PVE::Storage::LVMPlugin;" 2018-03-16 12:54:06 +01:00
Dietmar Maurer
fa1b42dd98 cifs plugin cleanups 2018-03-16 12:35:55 +01:00
Dietmar Maurer
c6fb7b80fc remove useless "use PVE::PTY;" 2018-03-16 12:09:37 +01:00
Wolfgang Link
c26f3a7105 Add read_password in CLI. 2018-03-16 12:08:24 +01:00
Wolfgang Link
6a26444462 Add cifsscan to CLI 2018-03-16 12:07:39 +01:00
Wolfgang Link
b9cfb8cac7 Add cifsscan to API 2018-03-16 12:07:14 +01:00
Wolfgang Link
4cab0acd27 Add cifsscan. 2018-03-16 12:06:37 +01:00
Wolfgang Link
d787523984 Register CIFSPlugin in the storage plugin system. 2018-03-16 12:06:14 +01:00
Wolfgang Link
ba7e9ba9c9 Add remove cifs in API call. 2018-03-16 12:05:02 +01:00
Wolfgang Link
a4a9405d76 Add cifs in create API call.
In this patch the nodes will be deleted if the nodes parameter comes with a empty string.
We need this in the GUI when update the nodes in the config to reset if a nodes.

If we do not erase the empty hash the storage online check would be skipped.
Also the password and user would not be verified.
2018-03-16 12:03:37 +01:00
Wolfgang Link
f79a699c13 Add set CIFS credentials. 2018-03-16 11:58:34 +01:00
Wolfgang Link
4792d439ad Add CIFS Storage Plugin.
This Plugin use as template the NFSpluigin.
We do only support smbversion 2 and 3.
Version 3 is default and must override through the config.
2018-03-16 11:57:57 +01:00
Wolfgang Bumiller
77445e9b80 bwlimit: apply limits when there's no override
Before, 'undef' was equivalent to unlimited, but '0' is the
"explicitly unlimited" value, so if the user doesn't request
an override, apply limits as if the user was unprivileged
(otherwise there's no way for privileged users to explicitly
ask to not override the configured limits).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-03-07 14:19:07 +01:00
Alwin Antreich
6ea1a3f3fe Fix typo in sub s/krdb_feature_disable/krbd_feature_disable
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-03-02 13:28:50 +01:00
Alwin Antreich
4c3b308583 Fix #1574: could not disable krbd-incompatible image features
To prevent an error when disabling features of a rbd image with already
disabled flags. This aborted the CT/VM cloning halfway through with
a leftover rbd image, but no vmid.conf to it.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-03-02 13:28:50 +01:00
Wolfgang Bumiller
9edb99a5a7 add Storage::get_bandwidth_limit helper
Takes an operation, an optional requested bandwidth
limit override, and a list of storages involved in the
operation and lowers the requested bandwidth against global
and storage-specific limits unless the user has permissions
to change those.
This means:
 * Global limits apply to all users without Sys.Modify on /
   (as they can change datacenter.cfg options via the API).
 * Storage specific limits apply to users without
   Datastore.Allocate access on /storage/X for any involved
   storage X.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-01-31 12:25:32 +01:00
Dominik Csapak
f6c03d367d whitespace cleanup
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-11-14 13:23:52 +01:00
Dominik Csapak
856c54bd75 add 'format' parameter to storage list call
we will use this for the gui to figure out if we have to show
a size selector, a file selector, which formats are there, etc.

we have to include this data even for not active storages, else
we cannot show the correct fields

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-11-14 13:23:52 +01:00
Dominik Csapak
5da48ca6ea add select_existing to plugindata for iscsi(direct)
this will be used in the gui, for determining if we need to select
something from the storage when using for an image

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-11-14 13:23:52 +01:00
Dominik Csapak
fc7c0e0504 (finally) fix #1099: return disklist correctly
in get_disks, when called with a parameter 'cciss/cXdY', we replaced
the '/' with '!' so that we can properly poll the information
about it from /sys/block/

but we have to replace the '!' with '/' again in our result list,
because the caller does not know anything about it and fails, because
the original dev is not in the list

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-11-14 13:23:52 +01:00
Wolfgang Link
8fe00d9944 Add logfung in storage_migrate
The log function is used by storage replication to redirect the output in the correct log.
2017-10-16 14:59:57 +02:00
Emmanuel Kasper
6b4acdb460 ISCSIPlugin: Use long options for readability
We otherwise use the long options everywhere in the plugin.
This will build the following command:

iscsiadm  --mode session --sid 1 --rescan
Rescanning session [sid: 1, target: xxx, portal: yyy]
2017-10-09 10:50:11 +02:00
Fabian Grünbichler
79127fb5c7 rbd: support auth=none if pveceph-managed
preserve the old behaviour of selecting auth_supported based on the
existence of the keyring, but limit it to external clusters.

this allows switching 'auth XXX required' in the pveceph-managed
ceph.conf while still automatically copying the keyring when adding a
storage.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-09-12 11:24:39 +02:00
Fabian Grünbichler
8143f49038 rbd: create ceph keyring dir before copying
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-09-07 12:51:31 +02:00
Fabian Grünbichler
5a39d0a111 rbd: manage keyring for pveceph storages
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-09-07 11:35:00 +02:00
Fabian Grünbichler
6eebc4a78a rbd: implement handling of unset monhost
add /etc/pve/ceph.conf to commands / option strings instead
of the monitor list provided via the 'monhost' option.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-09-07 11:35:00 +02:00
Fabian Grünbichler
239aa73e93 rbd: refactor cmdline helpers
these were line by line identical except for the binary path

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-09-07 11:35:00 +02:00
Fabian Grünbichler
0b9ef02ebc rbd: make monhost option optional
to allow differentiating between user-created external RBD storage
entries (WITH monhost), and those created and managed by pveceph
(without).

making monhost non-fixed allows easily opting into the managed behaviour via
'pvesm set STORAGE -delete monhost', but is also helpful for external clusters
(i.e., after adding or removing a monitor you need to update the monhost
parameter..)

adapt description accordingly.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-09-07 11:35:00 +02:00
Wolfgang Bumiller
de8eff4d31 dir: allow a path as is_mountpoint value
This turns is_mountpoint more into export(5)'s `mountpoint`
property.
Given the directory storage with the properties:

    path /a/b/c
    is_mountpoint $value

$value = yes
    Same as before, /a/b/c must be mounted.
$value = no (or not set)
    Same as before, no effect.
$value = /a/b
    New: /a/b must be mounted (as opposed to /a/b/c)
2017-09-06 09:26:20 +02:00
Wolfgang Bumiller
6ce4f72487 cleanup: call defined() early
No need to carry around a hash we don't need if we get one.
2017-08-10 13:24:41 +02:00
Philip Abernethy
b43b073b51 Fix #1474: Fixes input and output of print_storage
Accommodates changes in 44ae567 and d40e27d by
reordering checks to allow for proper filtering
of disabled storages. Also reorders two checks to
prevent autovivification resulting in disabled
storages always showing in output.
2017-08-10 13:20:52 +02:00
Dominik Csapak
e2bd817ccc add info about bluestore to disklist
this patch adds information about bluestore/db/wal to the disklist,
and we set the journal count only when we have at least one journal on
the disk

also adapt the regression tests

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-08-07 14:52:10 +02:00
Wolfgang Bumiller
0430101380 pvesm: remove superfluous space after N/A 2017-07-27 14:17:02 +02:00
Thomas Lamprecht
d40e27deb2 pvesm status: improve output and its format
Add column names at top of output, this allows easier understanding
of what each column means.

Use leading spaces on the percentage column so that this is lined up.

Switch out the 1/0 from the active column with the actual status
(active, inactive, disabled).

Show N/A if storage is disabled.

Use $res->{total} instead of calculating a sum of used and available.

Remove wrong rounding - if we want to display 2 digits from the
fractional part we would need to add 0.005 not 0.5, this made the
result quite wrong depending on the storage size.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2017-07-27 13:09:28 +02:00
Thomas Lamprecht
44ae567c38 storage_info: return also disabled storages
in the Storage/Status API call we have a 'enabled' param which had no
effect because storage_info only returned enabled one way or the
other.

This affected also `pvesm status` which uses the Storage/Status API
call.

So push also disabled storages to the info array but only activate
and get their status when thei are enabled.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2017-07-27 13:09:28 +02:00
Wolfgang Link
21aaefd522 fix ceph_version_parser
Ceph change ceph version output.

full output of 'ceph --version'

Luminous 'ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)'
Jewel 	 'ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)'
2017-07-04 06:10:15 +02:00
Emmanuel Kasper
e53050ed13 Fixes: #1415 do not block pvestatd when nfsd is stopped on the server side
Wrap the -d test with run_or_get_killed sub this test
can make pvestatd hang on I/O wait when a nfsd process is stopped

This might help with other file based storages, for instance
directory storages on unplugged USB devices.
2017-06-29 15:16:08 +02:00
Dietmar Maurer
7118dd916b PVE::Storage::storage_can_replicate - hew helper 2017-06-27 06:17:58 +02:00
Wolfgang Bumiller
6b3a4a2504 import/export: drop the 'subvol/' prefix from 'tar+size'
It was deemed an unnecessary overhead.

Also add a comment about the effect (or lack thereof) of
user namespace uid mappings.
2017-06-22 17:35:04 +02:00
Emmanuel Kasper
030f6e1a97 Fix #1424: allow tar.xz templates upload 2017-06-22 14:32:27 +02:00
Wolfgang Bumiller
766cfd9a87 import/export: use our standard tar flags
Custom plugins may reuse
@PVE::Storage::Plugin::COMMON_TAR_FLAGS as well.
2017-06-22 12:47:29 +02:00
Wolfgang Bumiller
e8a7e764af storage_migrate: with_snapshots parameter
Since the migration code already has to check snapshots and
we still want to be able to migrate raw disks, we cannot
always pass -with-snapshots 1.
2017-06-22 12:40:23 +02:00
Dietmar Maurer
e1f6cb3904 re-add changes to PVE/Storage/Plugin.pm (fixup previous commit)
This was removed by accident.
2017-06-22 09:05:29 +02:00
Dietmar Maurer
364e63647a bump version to 5.0-6 2017-06-22 08:36:11 +02:00
Wolfgang Bumiller
da72898cc6 migrate: only use import/export
This replaces the path-based and lvm/thin special cases in
storage_migrate with the already generic-enough zfspool
case which is already using import/export and does not
directly depend on zfs anymore.
2017-06-22 07:03:39 +02:00
Wolfgang Bumiller
5cbbc78fdb lvm: import/export: implement raw+size format 2017-06-22 06:48:09 +02:00
Wolfgang Bumiller
9559a62a3d import/export: new formats: raw, tar, qcow2, vmdk
All of them have a `+size` prefix to show that they're not
"pure raw" or "pure tar" streams, because some storage may
need to know in advance how much storage to allocate.
The formats are explained in comments.

PVE::Storage::Plugin now has default implementations for
these for non-incremental streams exporting the current
(rather than a snapshot state).
To use qcow2 or vmdk formats $with_snapshots must be true,
otherwise raw/tar will be used where $with_snapshots must
be false.
2017-06-22 05:59:34 +02:00
Wolfgang Bumiller
ae36189d26 export: add missing format query call parameter 2017-06-22 05:51:32 +02:00
Wolfgang Bumiller
488cf14d4f migrate: insecure: untaint the received ip/port 2017-06-09 12:30:59 +02:00
Wolfgang Bumiller
47cea19432 migrate: don't include the hostname in rsync's rsh command 2017-06-09 12:30:17 +02:00
Wolfgang Bumiller
4b4c580d02 migrate: use the correct IP with rsync 2017-06-09 12:30:02 +02:00
Dietmar Maurer
b43d0f3043 ZFSPoolPlugin.pm: remove unused code 2017-06-08 08:45:22 +02:00
Dietmar Maurer
636ac5b82f PVE::Storage::volume_snapshot_list - remove comment about ordering
Some storage types supports arbitrary snapshot trees, so there is
no strict ordering relation.
2017-06-07 06:36:55 +02:00
Dietmar Maurer
8b622c2dff PVE::Storage::volume_snapshot_list - remove $prefix parameter
Always return the full list of snapshots. Users of this library can easily
filter with a simply 'grep' instead.
2017-06-07 06:20:07 +02:00
Dietmar Maurer
c2bce0ef63 remove unused replication code (moved to pve-manager) 2017-06-07 06:14:30 +02:00
Dominik Csapak
a3f38a644c fix #1379: return size as number instead of string
this caused the webinterface to sort alphabetically instead of numerical
when sorting by image size

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-06-02 10:24:30 +02:00
Wolfgang Bumiller
7ba34faa6e migrate: add insecure parameter
This uses mtunnel's new -run-command parameter in
combination with the ssh-info's network cidr option to use
an unprotected TCP port.
2017-06-01 10:43:32 +02:00
Wolfgang Bumiller
01f7e902e9 migrate: implement rate limiting
Note that it does not apply to the local 'mkdir && cp' case.
2017-06-01 10:42:32 +02:00
Wolfgang Bumiller
cd0b3d5bc2 Revert "free: added snapshot parameter"
This reverts commit a26e1abc70.
2017-05-23 13:31:21 +02:00
Wolfgang Bumiller
b57a863a0f migrate: cleanup & fixup
The volume_snapshot call was missing the condition when to
create a snapshot. Make the whole logic easier to follow
with a $migration_snapshot boolean.
Also get rid of the remote `pvesm free -snapshot` call by
using import's new -delete-snapshot parameter.
2017-05-23 13:30:51 +02:00
Wolfgang Bumiller
5259593842 import: add -delete-snapshot parameter
This deletes a snapshot on *success*, done directly in the
CLI handler, as the rollback/delete on failure is already
happening inside the plugin's import method.
2017-05-23 13:29:48 +02:00
Wolfgang Bumiller
acd27197ce migrate: replace target ip parameter with sshinfo parameter 2017-05-23 09:31:19 +02:00
Wolfgang Bumiller
dc7695377c migrate: snapshot parameter for use with replication 2017-05-23 09:27:26 +02:00
Fabian Grünbichler
55f908a1cd migrate: cleanup: $free is always defined 2017-05-12 15:01:08 +02:00
Wolfgang Bumiller
7459cb3d91 migrate: query transfer stream formats 2017-05-12 14:42:17 +02:00
Wolfgang Bumiller
d390328bfd api: add import/export format querying 2017-05-12 14:42:17 +02:00
Wolfgang Bumiller
ac191ec740 migrate: snapshot parameter for incremental transfer 2017-05-12 14:42:17 +02:00
Wolfgang Bumiller
865710f47a migrate: use pvesm import/export for zfs
The only leftover zfs specific part here is the zfs format
type parameter.
2017-05-12 14:42:17 +02:00
Wolfgang Bumiller
a26e1abc70 free: added snapshot parameter 2017-05-12 14:42:16 +02:00
Wolfgang Bumiller
47f37b5362 pvesm: import/export commands 2017-05-12 14:42:16 +02:00
Dietmar Maurer
0d5a5fc9cb Revert "Include incremental zfs send in storage_migrate."
This reverts commit 4537523b5a.

We want to use send/receive for this - patch will follow.
2017-05-10 07:20:00 +02:00
Dietmar Maurer
f88846ea3b do not install replication code
The code introduces a cyclic package dependency, so we need to
move it to pve-manager package.
2017-05-10 07:06:35 +02:00
Dietmar Maurer
17be2e9a0c volume_snapshot_list: remove $ip parameter
We want to handle ssh connections somewhere else (not inside the
storage plugins).
2017-05-10 07:02:42 +02:00
Dietmar Maurer
3d4949692a Revert "Include new storage function volume_send."
This reverts commit b76774e57f.
2017-05-10 06:58:44 +02:00
Dietmar Maurer
44257d2e38 Revert "Add ip parameter in zfs_request to execute on remote host."
This reverts commit c4bb4a3d19.
2017-05-10 06:55:42 +02:00
Dietmar Maurer
889d7485cb Revert "Add function volume_snapshot_delete_remote."
This reverts commit 4bd0b38f53.
2017-05-10 06:55:00 +02:00
Dietmar Maurer
552278a2a2 PVE::ReplicationTools::update_conf - correctly catch exception 2017-05-09 11:45:12 +02:00
Dietmar Maurer
9998ea1d52 ReplicationTools.pm: call PVE::INotify::nodename inside methods
so that we can overwrite it using Test::MockModule. This makes writing
regression tests easier.
2017-05-09 07:11:07 +02:00
Dietmar Maurer
6971bcd61f ReplicationTools.pm_ do not pass $local_node to volume_snapshot_list 2017-05-09 07:08:17 +02:00
Dietmar Maurer
23ddb82e54 ReplicationTools.pm: rename $get_guestconfig to get_guest_config
make it a real sub
2017-05-09 06:36:24 +02:00
Dietmar Maurer
6a50f652a6 rename get_syncable_guestdisks to get_replicatable_volumes 2017-05-09 06:23:23 +02:00
Dietmar Maurer
666e0afebf ReplicationTools: use get_replicatable_volumes() 2017-05-09 06:20:24 +02:00
Dietmar Maurer
2971c957ab PVE::ReplicationTools::get_node_ip - do not pass noerr to PVE::Cluster::remote_node_ip()
We need an IP address.
2017-05-08 12:05:39 +02:00
Dietmar Maurer
83ba57c81c PVE::ReplicationTools - s/delet/delete/ 2017-05-08 10:22:57 +02:00
Dietmar Maurer
508344ab9d use consistent snapshot prefix for replicas 2017-05-08 09:29:57 +02:00
Dietmar Maurer
dfea1c130e PVE::ReplicationTools::destroy_all_snapshots - never delete local disks 2017-05-08 08:47:28 +02:00
Dietmar Maurer
be1d5bc192 PVE::ReplicationTools::sync_guest - do not add snapname to disks_status
Because it is not used at all.
2017-05-08 07:57:04 +02:00
Dietmar Maurer
c9a16105d4 PVE::ReplicationTools:sync_disk - use $lastsync instead of $job->{lastsync}
Because tghis is the parsed (untainted) value, and we pass this a parameter
to cleanup_snapshot.
2017-05-08 07:39:47 +02:00
Dietmar Maurer
81fa35e539 PVE::ReplicationTools::sync_guest - use integer compare for lastsync 2017-05-08 07:36:20 +02:00
Dietmar Maurer
ed283475d4 PVE::ReplicationTools::get_snapshot - remove dead code
This function makes no sense and is not used.
2017-05-08 06:55:12 +02:00
Dietmar Maurer
cba200746c PVE::ReplicationTools: new helper get_ssh_cmd() 2017-05-08 06:42:24 +02:00
Dietmar Maurer
8ef98169e3 PVE::ReplicationTools::cleanup_snapshot - make sure ip is defined 2017-05-08 06:28:58 +02:00
Dietmar Maurer
7e5b531380 use 'eq' instead of regex for simple equality test. 2017-05-06 17:20:52 +02:00
Dietmar Maurer
efcdabfb36 PVE::ReplicationTools - coding style cleanup 2017-05-05 17:48:29 +02:00
Dietmar Maurer
e4d5d5f2aa PVE::ReplicationTools::get_node_ip() - fix arguments for mtunnel 2017-05-05 15:45:27 +02:00
Dietmar Maurer
fa5e729eac PVE::ReplicationTools - add missing "use PVE::INotify;" 2017-05-05 15:38:52 +02:00
Dietmar Maurer
be304991e7 PVE::ReplicationTools - remove unused var $cluster_nodes 2017-05-05 15:36:01 +02:00
Dietmar Maurer
e9b759ac30 PVE::ReplicationTools - only use required modules (remove Time::Local) 2017-05-05 15:36:01 +02:00
Dietmar Maurer
d884a78870 PVE::ReplicationTools - remove useless call to cfs_update 2017-05-05 15:27:08 +02:00
Dietmar Maurer
8b38cdaf67 PVE::API2::Storage::Replication - add reasonable permission checks 2017-05-05 11:55:34 +02:00
Dietmar Maurer
f580b2d30d pvesr run: remove wrong permission check 2017-05-05 11:23:25 +02:00
Dietmar Maurer
1d3ddfaf2c pvesr: move destroy_job into API class, remove wrong permission checks. 2017-05-05 11:18:30 +02:00
Dietmar Maurer
339ca30a5b PVE::API2::Storage::Replication->jobs(): return array 2017-05-05 10:54:10 +02:00
Dietmar Maurer
e0992d57e1 pvesr: rename list => jobs, call API
to avoid code duplication. I also removed the functioanlity to query
jobs on other nodes, because it is clumsy and not really needed.
2017-05-05 10:31:51 +02:00
Dietmar Maurer
07dff26749 PVE::API2::Storage::Replication - add directory index 2017-05-05 10:01:26 +02:00
Dietmar Maurer
76c358149a renamed: PVE/API2/StorageReplication.pm -> PVE/API2/Storage/Replication.pm 2017-05-05 09:25:43 +02:00
Wolfgang Bumiller
6b1d6cee44 replica: get_syncable_disks: skip disks with replica=0 2017-04-28 11:31:46 +02:00
Wolfgang Bumiller
618a0d9b96 replica: add missing import 2017-04-28 11:27:59 +02:00
Wolfgang Bumiller
828d9073b3 replica: cleanup 2017-04-28 10:33:51 +02:00
Wolfgang Link
4537523b5a Include incremental zfs send in storage_migrate.
When replica is enabled and the target host is the reptarget,
the most VM data are on the new target.
2017-04-28 10:05:27 +02:00
Wolfgang Link
663510b86d This patch will include storage asynchronous replication.
It is possible to synchronise a volume to an other node in a defined interval.
So if a node fail there will be an copy of the volumes from a VM
on an other node.
With this copy it is possible to start the VM on this node.
2017-04-28 10:05:27 +02:00
Wolfgang Link
f189504ccb Add replicate as new storage feature.
This feature shows that the storage can send and receive images.
2017-04-28 10:05:27 +02:00
Wolfgang Link
4bd0b38f53 Add function volume_snapshot_delete_remote.
This function we need for replica to handle snapshots on remote nodes.
2017-04-28 10:05:27 +02:00
Wolfgang Link
c4bb4a3d19 Add ip parameter in zfs_request to execute on remote host.
We need this function to delete remote snapshots.
2017-04-28 10:05:27 +02:00
Wolfgang Link
aefe82ea03 Include new storage function volume_snapshot_list.
Returns a list of snapshots (youngest snap first) form a given volid.
It is possible to use a prefix to filter the list.
2017-04-28 10:05:27 +02:00
Wolfgang Link
b76774e57f Include new storage function volume_send.
If the storage backend support import and export
we can send the contend to a remote host.
2017-04-28 10:05:27 +02:00
Fabian Grünbichler
fc1089fc3a fix run_command return code handling 2017-04-10 15:29:29 +02:00
Fabian Grünbichler
c59a0f2452 rbd: fix rados df parser for luminous 2017-03-21 10:13:02 +01:00
Dominik Csapak
f5451f288d remove immutable flag also for subvols on directory storage
or else the removal of such templates (with rootfs size 0) fails

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-03-10 09:21:53 +01:00
Fabian Grünbichler
d746da86d8 rbd: fix exit from 'rbd ls -l' parser 2017-02-23 09:22:57 +01:00
Wolfgang Bumiller
b2e430d30f rbd: minor regex fixup 2017-02-20 10:00:57 +01:00
Fabian Grünbichler
292a33fdd4 rbd: fix typo
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-02-13 12:13:06 +01:00
Fabian Grünbichler
4db25fa2a6 rbd: use consistent image name schemes
since we allow vm-ID-whatever when allocating images, we
should also include those when listing them.

note: '@' is reserved for snapshots in ceph, so it is safe to
skip lines including an '@' in the image name.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-02-13 12:13:06 +01:00
Fabian Grünbichler
53a236f2c5 rbd: use 'rbd ls' without '-l' to find free names
with more than a few images, 'rbd ls -l' gets rather slow
compared to a simple 'rbd ls'. since we only need to check
existing image names for finding a free one, the latter is
sufficient.

example with ~400 rbd images:
$ time rbd ls -p ceph-vm > /dev/null
real    0m0.027s
user    0m0.012s
sys     0m0.008s

$ time rbd ls -l -p ceph-vm > /dev/null
real    0m5.250s
user    0m1.632s
sys     0m0.584s

a linked clone of two disks on the same setup accordingly
also shows a massive speedup:

$ time qm clone 1000 10000 -snap test
create linked clone of drive scsi0 (ceph-vm:vm-1000-disk-2)
clone vm-1000-disk-2: vm-1000-disk-2 snapname test to
vm-10000-disk-1
create linked clone of drive scsi1 (ceph-vm:vm-1000-disk-1)
clone vm-1000-disk-1: vm-1000-disk-1 snapname test to
vm-10000-disk-2

real    0m11.157s
user    0m3.752s
sys     0m1.308s

$ time qm clone 1000 10000 -snap test
create linked clone of drive scsi1 (ceph-vm:vm-1000-disk-1)
clone vm-1000-disk-1: vm-1000-disk-1 snapname test to
vm-10000-disk-1
create linked clone of drive scsi0 (ceph-vm:vm-1000-disk-2)
clone vm-1000-disk-2: vm-1000-disk-2 snapname test to
vm-10000-disk-2

real    0m0.872s
user    0m0.652s
sys     0m0.096s

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-02-13 12:13:06 +01:00
Dietmar Maurer
ec73c0ff67 fix bug #1280: Can't locate object method "root@pam"
This was introduced by commit 04a13668b9.
2017-02-12 09:55:50 +01:00
Fabian Grünbichler
086c4bf12b extract_vzdump_config: handle RE consistently
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-01-20 11:14:01 +01:00
Fabian Grünbichler
8944339466 weaken matching for vzdump config extract filename
to allow "pvesm extractconfig filename" when in the dump
directory.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-01-20 11:13:58 +01:00
Dietmar Maurer
04a13668b9 PVE::Storage: new helper check_volume_access()
Copied from PVE::RPCEnvironment to avoid cyclic dependency
(pve-storgae => pve-access-control => pve-storage).
2017-01-19 09:14:41 +01:00
Dietmar Maurer
ef881e10eb fixes for new PVE::RPCEnvironment implementation
Use PVE::RPCEnvironment->is_worker() instead of
PVE::RPCEnvironment::is_worker().
2017-01-19 09:14:41 +01:00
Wolfgang Bumiller
e438d0940f fix #1252: rbd: delete snapshots when using krbd 2017-01-17 10:32:23 +01:00
Dmitry Petuhov
a4aee43380 Fix RBD resize with krbd option enabled.
With krbd we resize volume and tell QemuSever to notify running QEMU
with zero $size by returning undef.

Signed-off-by: Dmitry Petuhov <mityapetuhov@gmail.com>
2017-01-16 09:14:00 +01:00
Dietmar Maurer
f984732e0e add setup_environment hook to CLIHandler class
and use run_cli_handler instead of depreciated run_cli.
2017-01-12 13:27:43 +01:00
Alexandre Derumier
e2e74320b7 sheepdog : volume_resize return if running
fix : https://forum.proxmox.com/threads/sheepdog-disk-resize-not-working.31760/

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
2017-01-09 13:27:11 +01:00
Dominik Csapak
1590fc13bf fix #1099 (again): map cciss/c0d0 to cciss!c0d0
there was still a point where we got the wrong string
on createosd we get the devpath (/dev/cciss/c0d0)

but need the info from get_disks, which looks in /sys/block
where it needs to be cciss!c0d0

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2017-01-05 15:54:11 +01:00
Fabian Grünbichler
afd0afe4f5 fix #1135: always clean up ZFS snapshots on migration
otherwise there are situations where snapshots are left
behind for already sent volumes. also include more warnings.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2017-01-03 11:45:16 +01:00
Dietmar Maurer
46f114a53f storage_migrate: use bs=4k to copy images with dd
The default block size is 512 bytes, and is quite slow. We do not
want to use blocks bigger than 4k, because we want to detect empty
blocks (sparse).
2016-12-24 10:55:31 +01:00
Wolfgang Link
0d28307d2f Add function get_blockdev.
This function will return you the block device of a given partition path.
2016-12-22 12:14:34 +01:00
Wolfgang Link
3196c387d2 Add new function part_num
With this function you get the partnum of a dev.
2016-12-22 12:12:47 +01:00
Dominik Csapak
7b32e2a7cd add crucial smart attribute for wear leveling
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-11-30 17:20:09 +01:00
Dominik Csapak
bd54091c2a catch '---' in threshold output of sandisk ssds
sandisk ssds have a default threshold of '---' on nearly all fields,
which prevents our parsing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-11-30 17:19:43 +01:00
Dominik Csapak
2ff3b506a1 use qemu gluster blockdriver for linked clone creation
this works around a bug, where qemu does not align the qcow2 file
when using the filesystem directly, and the gluster blockdriver
refuses to read from it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-11-29 09:40:10 +01:00
Fabian Grünbichler
72bdeea1bf increase default timeout for zpool import
as zpool import can easily take longer than 5 seconds on
systems with lots of disks
2016-11-29 09:30:26 +01:00
Fabian Grünbichler
e2e6380112 improve zpool activate_storage
the old code was way too broad here, this fixes at least the
following issues:
- importing of other/unconfigured zpools by "import -a"
- possible false positives if a pool name is a substring of
  another pool name because of "list" without pool name,
  potentially skipping activation for such pools
- not noticing failure to activate in activate_storage
  because the success of "zpool import -a" does not tell us
  anything about the pool we actually wanted to import

checking specifically for the pool to be activated when
calling "zpool list" gets rid of the second issue, and
trying to import only that pool fixes the other two.
2016-11-29 09:29:33 +01:00
Stefan Priebe
4b7dd9d743 allow --allow-shrink on RBD resize
Signed-off-by: Stefan Priebe <s.priebe@profihost.ag>
2016-11-17 09:06:55 +01:00
Dominik Csapak
571b6f2638 add default rotational value
because if the file does not exist,
we have an perl error for comparing an uninitialized
value

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-28 10:20:28 +02:00
Dominik Csapak
865bdbd917 use model from udevadm
we want this, because the model in /sys/block/<device>/device/model
is limited to 16 characters

and since the model is not always in the udevadm output (nvme),
also read the model from the model file as fallback

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-28 10:19:40 +02:00
Dominik Csapak
10a48db55c make dir_is_empty a proper sub
this allows us later to mock the sub,
which we need for testing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-28 10:19:40 +02:00
Dominik Csapak
461a9fd8da move directory test into get_sysdir_info
because it logically belongs there, also
this makes the testing easier

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-28 10:19:40 +02:00
Dominik Csapak
532e89e7be use /sys/block/ path for udev instead of name
since we iterate over the entries in /sys/block
it makes sense to use this path

this should fix #1099
because udevadm does not take
 -n cciss!c0d0 (because it only looks in dev for this)
but takes
 -p /sys/block/cciss!c0d0

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-28 10:19:40 +02:00
Fabian Grünbichler
8b5ccc06b7 fix #1165: only check mount status when is_mountpoint is set
otherwise the status() method returns garbage for non-mount
point directory storages.
2016-10-13 08:29:30 +02:00
Wolfgang Bumiller
d57fb43d9b disks: fix warnings 2016-10-07 09:28:06 +02:00
Dominik Csapak
a0028cf97e allow rbd images < 1M to be detected
without this, having an efidisk on a ceph storage
prevents creating another disk on the same
ceph storage, because it will not be detected
and we try to allocate one with the same name

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-07 08:17:40 +02:00
Dominik Csapak
38ddd4ce8e correctly parse cciss devices in get_disks
cciss maps the devices to
/sys/block/cciss!cXdY

and not to

/sys/block/cXdY

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-06 07:50:05 +02:00
Dominik Csapak
6965a6701e fix #1135: refactor wear level parsing
refactored the wear level parsing into its
own function, where we can now define a
vendor <-> attribute id
mapping

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-06 07:49:30 +02:00
Dominik Csapak
dd902da78e merge get_smart_data/health
instead of parsing the output of smart in two places,
give get_smart_data a flag if we only want health

this fixes a bug (not on the bugtracker), where
an ssd with disabled smart had an empty string as health
in the gui

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-10-06 07:48:57 +02:00
Fabian Grünbichler
c9bd3d22b2 fix #1123: modify NVME device path for SMART support
seems like we need to drop the namespace part before calling
smartctl.

note that NVME devices require smartmontools >= 6.5
2016-10-04 10:35:30 +02:00
Fabian Grünbichler
16bf963b94 add return schema to smart API path 2016-10-04 10:34:47 +02:00
Fabian Grünbichler
dc1311cbe8 fix #1120: non-ATA disks with unstructured smart data
simple parser as fallback for non-ATA disks, just collect
all lines in the "SMART DATA" section.
2016-10-04 10:34:22 +02:00
Fabian Grünbichler
5db2d529ed fix #1120: SCSI SMART health parsing
SCSI disks report their health as "SMART Health Status: OK"
2016-10-04 10:34:06 +02:00
Fabian Grünbichler
7a98a62dbc disks: allow get_disks without smart checks
the smart checks are only needed for the API call(s) that
list all disks and their status, but get_disks is also used
in disk usage checks and in the Ceph code, where the smart
status is completely irrelevant.

drop the implicit skipping of smart checks if $disk is set,
since we have an explicit parameter for this now.
2016-10-04 10:33:49 +02:00
Fabian Grünbichler
1993540bf8 fix #1122: correctly create LUNs for linked clones 2016-09-29 08:42:26 +02:00
Fabian Grünbichler
e3b02ffe6e disks: fix typo 2016-09-29 08:42:15 +02:00
Fabian Grünbichler
1c99955364 disks: parse smart attributes using RE 2016-09-29 08:42:06 +02:00
Fabian Grünbichler
0c486b09df disks: use smartctl -H -A
to only list SMART health and attributes, instead of
"smartctl -a", which prints "all SMART information"
2016-09-29 08:41:31 +02:00
Fabian Grünbichler
acd3d91649 move SMART error handling into get_disks
because we never ever want to die in get_disks because of a
single disk, but the nodes/xyz/disks/smart API path is
allowed to fail if a disk device is unsupported by smartctl
or something else goes wrong.
2016-09-29 08:40:19 +02:00
Wolfgang Bumiller
d547f26c7d Fix #1012: dir storage: add is_mountpoint option
While the mkdir option deals with the case where we don't
want to clobber a mount point with directories (like ZFS,
gluster or NFS), putting a directory storage directly onto a
mount point is still risky:
If the path exists - which it usually does even if not
mounted - the storage will be considered successfully
activated, but empty (or with unexpected content). Some
operations will then lead to unexpected problems: the
free_disk operation for instance only warns if the disk does
not exist, but does not throw an error. In this case the
configuration might be updated without the real disk being
deleted. Once it's mounted back in, later operations which
check existing disks which are not part of the current VM
configuration (like migration) might error unexpectedly.

This adds an 'is_mountpoint' option to directory storages
which assumes the directory is an externally managed mount
point (eg. fstab or zfs) and changes activate_storage() to
throw an error if the path is not mounted.
2016-09-27 09:56:55 +02:00
Wolfgang Bumiller
c7616abcb2 path based storages: improve the mkdir option
So far this only prevented the creation of the toplevel
directory. This does not cover all problem cases,
particularly when said directory is supposed to be a mount
point, including NFS and glusterfs beside ZFS.

The directory based storages we have already use mkpath
whenever they need to create files, and for actions on files
which are supposed to exist it's fine if it errors out.
So it should also be safe to skip the creation of standard
subdirectories in activate_storage().

Additionally NFS and glusterfs storages should also accept
the mkdir option as they otherwise may exhibit similar
issues, eg. when an NFS storage is mounted onto a directory
inside a ZFS subvolume.
2016-09-27 09:54:53 +02:00
Dietmar Maurer
ff3badd83f white space cleanups 2016-09-26 13:40:43 +02:00
Wolfgang Link
a9ef8ffb16 Avoid JavaScript gets a string 0.
If the JavaScript gets a "0" it convents it to a boolean false.
So to ensure the GUI always get valid int we cast the values.
2016-09-26 13:38:56 +02:00
Fabian Grünbichler
787624dfc0 add comments about LVM thin clones 2016-09-15 14:01:07 +02:00
Fabian Grünbichler
5510f5c9f9 fix typo 2016-09-15 13:56:17 +02:00
Fabian Grünbichler
1b83c3d9c7 harmonize list_images code 2016-09-15 13:54:47 +02:00
Fabian Grünbichler
17fb7e4215 move check for existing clones into own method
and change its return type to boolean
2016-09-15 13:52:57 +02:00
Fabian Grünbichler
9924228be1 remove unused method
only used by test case, which should use what the rest of
the codebase uses as well
2016-09-15 13:42:55 +02:00
Dietmar Maurer
cfd58f1fcc code cleanup 2016-09-14 11:31:10 +02:00
Fabian Grünbichler
3718e83ab5 fix error message 2016-09-14 11:24:06 +02:00
Fabian Grünbichler
883d9b81f0 fix indentation 2016-09-14 11:23:52 +02:00
Fabian Grünbichler
9690e55e9b rbd: detect linked clones/base volumes correctly
since the rbd images themselves are named differently than
the volumes in our config files, we need to recreate this
information from the parent relation in the ceph metadata,
otherwise list_images() might return wrong volume names/IDs

since list_images is used by PVE::Storage::vdisk_free() to
check for children still referencing a base image, because
of the wrong volume id RBDPlugin->parse_volname() does not
detect the base image of linked clones and the check fails.
this is thankfully mitigated by the protected status of the
base snapshot, but creates a rather confusing error message.

scenario (VM 701 is a linked clone of template VM 700):

$ qm config 700 | grep virtio0:
virtio0: ceph_qemu:base-700-disk-1,size=2G
$ qm config 701 | grep virtio0:
virtio0: ceph_qemu:base-700-disk-1/vm-701-disk-1,size=2G

before (pvesm list reports wrong volume ID, check fails):

$ pvesm list ceph_qemu
ceph_qemu:base-700-disk-1   raw 2147483648 700
ceph_qemu:vm-701-disk-1     raw 2147483648 701
$ pvesm free ceph_qemu:base-700-disk-1
snap_unprotect: can't unprotect; at least 1 child(ren) in pool rbd
rbd unprotect base-700-disk-1 snap '__base__' error: snap_unprotect: can't unprotect; at least 1 child(ren) in pool rbd

after (correct volume ID, check works as intended):

$ pvesm list ceph_qemu
ceph_qemu:base-700-disk-1                   raw 2147483648 700
ceph_qemu:base-700-disk-1/vm-701-disk-1     raw 2147483648 701
$ pvesm free ceph_qemu:base-700-disk-1
base volume 'base-700-disk-1' is still in use (use by 'base-700-disk-1/vm-701-disk-1')
2016-09-14 11:23:39 +02:00
Fabian Grünbichler
5e6aa346c7 rbd: use correct key to access hash elements 2016-09-14 11:17:16 +02:00
Dominik Csapak
9018a4e639 do not automatically die on smartctl exit code > 0
since smartctl uses the return value to encode
disk health status (such as failure in the past)
we cannot die there, but have to parse the returncode

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-09-08 16:52:33 +02:00
Dietmar Maurer
2622a5ca2d sheepdog 1.0 changed the path from /usr/sbin/dog to /usr/bin/dog 2016-09-07 09:11:43 +02:00
Dominik Csapak
409f8203e0 add api entries for disk management
adds a new class (intended to be used under nodes in pve-manager)
which adds the three api calls: list, smart and init

list being a general list of the available disk with infos
smart being a call to get the smart data from a given device
init being a call to write a gpt header to an unused disk

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-09-05 13:49:28 +02:00
Dominik Csapak
cbba9b5b9c add Diskmanage Utilities
this adds the functions for listing the disks (mostly copied from
the ceph code), checking if a disk is a valid blockdevice, if it
is used/in a zfs pool/as an lvm pv, and an init function (just to add a gpt header;
this is important if one wants to use a fresh disk for ceph journals)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-09-05 11:31:19 +02:00
Dietmar Maurer
6d2b278c51 white space cleanups 2016-09-01 06:28:54 +02:00
Dietmar Maurer
e7ac2d5cf6 rbd_unittobytes: use a local var instead of a sub 2016-09-01 06:24:51 +02:00
Alexandre Derumier
134172255f rbd: allow to use client custom ceph conf for each storeid
If you want to use different ceph storage,
something they have differents values like ms_nocrc = true.(they are also others ones).

The client need to specify theses special options to be able to connect

This patch allow to create a ceph config file for each storeid in

/etc/pve/priv/ceph/$storeid.conf

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
2016-09-01 06:21:40 +02:00
Dmitry Petuhov
4dee23d305 Add support for custom storage plugins
PVE team cannot support specialized vendor-specific storage
plugins because of lack of hardware. But we can allow users to
add own plugins for their storages without need to rewrite any
PVE code and thus ease PVE updates to them.

Idea of this patch is to add folder /usr/share/perl5/PVE/Storage/Custom
where user can place his plugins and PVE will automatically load
them on start or warn if it could not and continue. Maybe we could
even load all plugins (except PVE::Storage::Plugin itself) this way,
because current storage plugins are not really plugins, if they
need to be explicitly loaded in PVE code :-).

Custom plugins MUST have api() method returning version for which
it was designed. If API changes from PVE side, module is just not
being registered and warnig message is printed do log, so user have
to update module. Until module update, corresponding storage will
just disappear from PVE, so it shall not impose any data damage
because of API change.

This approach works (with some limitations) if plugin works in
generic PVE way: full control of volumes lifecycle. And will not
currently work for custom plugins like iSCSI, which needs to select
pre-existing volumes. Maybe someone will add more flexible way to
pve-manager to select input elements for storage plugins to target
this.

Currently tested with my NetApp plugin.

Signed-off-by: Dmitry Petuhov <mityapetuhov@gmail.com>
2016-08-26 15:44:49 +02:00
Dominik Csapak
f3b3b2a3b7 remove compression option from lvm migration
ssh(1) mentions that compression is only disirable on slow
connections.

since migration from cluster node to cluster node needs a
fast network anyway, we can drop the compression for
a speed improvement

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-08-05 08:36:13 +02:00
Fabian Grünbichler
82fc923fd4 fix spelling / grammar 2016-07-13 13:59:26 +02:00
Wolfgang Bumiller
e6ccfdeb21 Remove unused pve-storage-monhost format 2016-07-11 13:58:56 +02:00
Wolfgang Bumiller
e858048fc5 rbd: use pve-storage-portal-dns-list for monhost
This way we get parameter verification on monitor addresses
as well as the ability to pass multiple `--monhost`
arguments to `pvesm add`.

Since our '-list' schemas default to using commas we now
need to properly support these, so all uses of the monhost
property now replace all of semicolon, space or comma into
the currently required character.
This should fix the issues reported by Alwin Antreich on the
pve-user list.

Since this schema supports both ipv6+port notations we need
to make sure we convert to the bracket enclosed variant.
Added a helper for this.
2016-07-11 13:58:56 +02:00
Wolfgang Bumiller
33cef4c84e rbd: path: don't build the entire path if we don't use it 2016-07-11 13:58:56 +02:00
Fabian Grünbichler
0423e8c686 fix indentation 2016-06-29 11:42:03 +02:00
Fabian Grünbichler
7a9dd1195d add tagged_only option to LVM storage
to filter volumes by the 'pve-vm-ID' tag, which is set on
all volumes created via the PVE storage layer.
2016-06-29 11:42:03 +02:00
Wolfgang Link
4a7d222204 add check if format is defined to avoid warning 2016-06-28 11:40:26 +02:00
Wolfgang Link
966ecef2e8 fix #1033 storage_migrate on LVMThin - add die.
This is necessary to ensure the process will proper finished.
2016-06-17 14:55:38 +02:00
Wolfgang Link
e967e0ef27 fix #1022 correct typo 2016-06-14 12:32:26 +02:00
Fabian Grünbichler
e76dbd9204 use correct ceph version command
"ceph version" retrieves the version from the cluster (i.e.,
from the queried monitor), but what is needed here is the
local ceph version, which is returned by "ceph --version".
2016-06-14 11:46:32 +02:00
Wolfgang Bumiller
5c95e48479 Dir storage creation: check for a sane path
Ideally we don't need this, but this with the directory
storage this is a user-input field which gets returned
by the storage's path() method which is used in various
external command calls.
2016-06-09 18:15:28 +02:00
Wolfgang Bumiller
602eacfe6a split udevam command call 2016-06-09 18:15:09 +02:00
Wolfgang Bumiller
38e1eb3d87 upload: whitelist filename chars instead of blacklisting 2016-06-09 18:14:55 +02:00
Wolfgang Link
e83f7b4095 fix for storage migrate
remove the sparse parameter from dd.
This destroy the lvm volume.

also remove the free_disk this will later be done.
2016-06-08 16:35:27 +02:00
Dietmar Maurer
84252f6772 code cleanups 2016-06-08 09:20:11 +02:00
Dietmar Maurer
76f13c9ee8 cleanup: improve error message 2016-06-08 08:36:18 +02:00
Wolfgang Link
0a29ad6148 Add LVM-thin storage migration
It is now possibel to migrate LVM-thin volumes offline from one node to an other.
Also LVM what is used by Qemu.
2016-06-08 08:21:40 +02:00
Wolfgang Bumiller
b521247bf4 fix 1012: dir: add mkdir option
By default a directory storage creates its path. In some
cases this can be undesired, mostly when storages have
nested paths (eg. a dir storage on a ZFS path or in an NFS
share, or inside custom mount points).
As a simple fix to this the 'mkdir' option (default ON)
can now be used to disable this behavior.
2016-06-07 10:47:16 +02:00
Dietmar Maurer
0be02e0f21 s/version_parser/ceph_version_parser/ 2016-06-07 10:32:42 +02:00
Fabian Grünbichler
d86fd0a49b disable jewel image features when using krdb
otherwise mapping those images will fail. disabling the
features only needs to be done once per image, so it makes
sense to do this when creating the images.

unfortunately, the command does not work in hammer, so
it needs a version check for jewel or higher.
2016-06-07 10:25:36 +02:00
Fabian Grünbichler
7aeda03306 add ceph version helpers 2016-06-07 10:25:25 +02:00
Fabian Grünbichler
fa017b9696 add pvesm extractconfig 2016-06-03 12:04:59 +02:00
Fabian Grünbichler
8898dd7be2 add extract vzdump config methods
extract_vzdump_config_tar is an adapted combination
of tar_archive_search_conf() and the first part of
recover_config(), both from PVE::LXC::Create.

a compressed vma backup file needs special error
handling because vma exits as soon as it found the config
file, which the used decompressors treat as error.
2016-06-03 12:04:26 +02:00
Fabian Grünbichler
daccf21ef7 docs: typo, newlines, cleanup 2016-04-15 16:37:01 +02:00
Dietmar Maurer
9e4632c2fe DRBDPlugin: check_drbd_res() ignore info codes
Messages for return codes 1 to 99 are not considered an error.
2016-04-15 08:07:01 +02:00
Wolfgang Bumiller
72e743bd65 rbd: fix error message 2016-04-11 12:57:51 +02:00
Dietmar Maurer
dbf890f082 generate manpage with pve-doc-generator, bump version to 4.0-49 2016-04-06 11:28:09 +02:00
Wolfgang Link
74b724a699 In path use parsed volname not the volid and add 'basevol' 2016-04-05 15:43:02 +02:00
Dietmar Maurer
8e5b96cac3 zfs_parse_zvol_list: simplify regex 2016-04-04 08:51:13 +02:00
Wolfgang Link
851658c3b0 Change zfs path when link clone are used
The new naming is coherent to Dir plugin.

So if we make an licked clone the parent will be coded in the path of the storage.
2016-04-04 06:38:15 +02:00
Wolfgang Link
703de49ea9 Skip invalid property's in storage parser
There is no need to remove the hole storage, if one property is not valid.
Just ignore the property.
2016-04-04 06:30:53 +02:00
Fabian Grünbichler
83d7192ff9 Add write_config, drop cfs_read_file
Use PVE::Storage::config() and the new
PVE::Storage::write_config() instead of cfs_read_file and
cfs_write_file with a hardcoded filename.
2016-03-30 10:42:12 +02:00
Wolfgang Link
986303caf3 Fix LXC migration from local subvol
This patch reconfigured the rsync parameters, so the fs keeps all settings and works recursive.
2016-03-22 09:06:59 +01:00
Wolfgang Link
b650f029d3 Use array to run_command instead of a string. 2016-03-17 07:07:56 +01:00
Wolfgang Link
2d22db9a99 Include all snapshots when migrate storage zfs to zfs. 2016-03-17 07:07:01 +01:00
Fabian Grünbichler
21430e5088 Use is_worker to decide default timeout for ZFS
Bump timeout to 1 hour if running in a worker and no timeout
specified.
2016-03-15 16:50:00 +01:00
Wolfgang Bumiller
f44e50fed1 lvmthin: activate base volumes
create_base() uses '-ky' to prevent base images from being
activated by default, similar to snapshots. This means we
need to activate them like snapshots with the '-K' option.
2016-03-15 06:48:59 +01:00
Wolfgang Link
7a047fce4a Remove content type container from GlusterFS. 2016-03-02 17:14:07 +01:00
Dominik Csapak
baafddbd02 add sparseinit to has_feature
we will use this for determining
if we need to write zeros to a volume

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-02-24 17:16:25 +01:00
Dietmar Maurer
668f6d9faf add lvmthinscan to CLI 2016-02-20 10:11:59 +01:00
Dominik Csapak
d0ea89e564 prepare storage for lvmthin gui
this patch adds an lvmthin scan to the api, so that we can get a list
of thinpools for a specific vg via an api call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2016-02-20 09:51:12 +01:00
Wolfgang Bumiller
2d080bb88d storage_migrate: another ipv6 fix for an rsync command 2016-01-28 11:10:56 +01:00
Wolfgang Link
19e5596a74 increase timeout for ZFSPugin
This is useful on large zfs pools because they take longer to response.
2016-01-25 10:48:16 +01:00
Dietmar Maurer
030bc5c803 lvmthin: allow to clone from snapshot 2016-01-20 11:32:18 +01:00
Dietmar Maurer
920ecf563a remove option maxfiles from zfspool plugin
It makse no sense, because this plugin is unable to store backup files.
2015-12-30 17:16:37 +01:00
Wolfgang Bumiller
1773e785c2 nfs: is_mounted: match /^nfs.*/ type
This is consistent with the old behavior.
2015-12-09 16:15:31 +01:00
Wolfgang Bumiller
aed6c85d28 nfs/glusterfs: is_mounted fixes
The parse_proc_mounts change made the glusterfs is_mounted
check fail (causing it to be shown as inactive on the GUI).
The NFS check was stricter (not allowing a trailing / in the
source anymore).
2015-12-09 09:22:15 +01:00
Dietmar Maurer
f482231e48 Revert "Change zfspoolplugin path when snapshot is given."
This reverts commit fdd31ce759.

The assumption was wrong. Turned out that we cannot assume
snapshots are always mounted there.
2015-12-09 07:38:36 +01:00
Wolfgang Bumiller
1f5734bb8d allow rx permissions for group/world on .subvol dirs
vdisk_alloc comes in with an umask of 0037, which means the
.subvol dir has permissions 0740, which means that the root
directory of containers has permissions 0740, essentially
preventing the users inside a container from accessing
anything.
2015-11-26 12:04:59 +01:00
Wolfgang Link
fdd31ce759 Change zfspoolplugin path when snapshot is given.
Zfs subvol snapshots are always mounted under $path_of_subvol/.zfs/$snapname
2015-11-19 12:36:19 +01:00
Phillip Schichtel
281f958706 Fixed ZFS over iSCSI snapshot rollback
I converted several zfs_request($class, ...) calls to $class->zfs_request(...) calls in ZFSPoolPlugin.pm and removed a superfluous $class parameter in ZFSPlugin.pm.

Fixes #816

Signed-off-by: Phillip Schichtel <phillip.public@schich.tel>
2015-11-18 11:00:40 +01:00
Wolfgang Bumiller
80b647882e make use of the new ProcFSTools::parse_proc_mounts 2015-11-14 10:37:06 +01:00
Dietmar Maurer
30650d3c46 lvmthin: enable copy from snapshot feature 2015-11-12 11:15:37 +01:00
Dietmar Maurer
33adbdc447 lvmthin: allow to acces snapshots in filesystem_path 2015-11-12 11:14:47 +01:00
Dietmar Maurer
faabe9e297 lvmthin: fix used space (only use data%) 2015-11-12 09:47:49 +01:00
Dietmar Maurer
f1b59efc0d lvmthin: implement template/clone 2015-11-12 09:31:45 +01:00
Dietmar Maurer
b13784612d new helper lvm_find_free_diskname() 2015-11-12 09:29:00 +01:00
Dietmar Maurer
5b41084a6a lvmthin alloc_image: add additional test if VG exists 2015-11-12 08:02:50 +01:00
Dietmar Maurer
663372bc4d lvmthin: implement snapshot and rollback 2015-11-12 07:28:28 +01:00
Dietmar Maurer
a9f0c6c4f5 lvmthin volume_resize: reuse code from parent class 2015-11-11 10:18:44 +01:00
Dietmar Maurer
5c687bd921 white space cleanups 2015-11-11 09:17:31 +01:00
Dietmar Maurer
15334c831c LvmThinPlugin: improve basic functionality, better code reuse 2015-11-11 09:14:06 +01:00
Dietmar Maurer
3e44cd8498 implement generic lvm_list_volumes()
This replace lvm_lvs, and return a list of all volumes. That way
we can reuse code for LvmThin.
2015-11-11 09:14:06 +01:00
Dietmar Maurer
610798bc2d add lvm thin plugin 2015-11-11 09:14:06 +01:00
Dietmar Maurer
5dca5c7c7e move format registration to Plugin.pm 2015-11-11 09:13:57 +01:00
Dietmar Maurer
efaf4017f4 zfs: enable posixacl and xattr for subvolumes 2015-11-06 16:39:17 +01:00
Wolfgang Link
27a2764632 fix bug #809: inheritance from ZFSPoolPlugin is not possible because we have to handle there subvols too. 2015-11-06 08:34:49 +01:00
Dietmar Maurer
bb3fc38b2d ISCSIPlugin: improve session parser 2015-11-02 17:12:14 +01:00
Wolfgang Link
0589e5f933 insert the format type at volume_resize
qemu-img needs now the format to resize image.
2015-10-28 11:30:05 +01:00
Wolfgang Bumiller
323ab5baa8 fix bug #769: migrate: use -p on 'zfs send' to include properties 2015-10-15 10:33:34 +02:00
Philipp Marek
0bdf560cff DRBD: Fix typo in comment. 2015-10-15 08:54:48 +02:00
Philipp Marek
046fd4cb61 DRBD: Remove search loop, limit results via the query. 2015-10-15 08:53:08 +02:00
Dietmar Maurer
dc6ff39f57 allow to choose content type for local storage
Also allow to disable local storage.
2015-10-12 06:52:28 +02:00
Dietmar Maurer
79f2b938d4 zfspool: fix volume_size_info and volume_resize for subvols 2015-10-08 09:49:36 +02:00
Dietmar Maurer
68b2c18acd allow containers on lvm 2015-10-01 13:02:13 +02:00
Dietmar Maurer
b70b0c58b3 complete_volume: avoid large list 2015-10-01 10:56:19 +02:00
Dietmar Maurer
f3bd890d98 add missing completion hooks 2015-10-01 07:59:48 +02:00
Dietmar Maurer
bf7aed2674 implement complete_volume 2015-10-01 07:59:48 +02:00
Dietmar Maurer
98437f4c85 add bash completion for content type 2015-10-01 07:16:59 +02:00
Dietmar Maurer
37ba0aea5b volume_list: moved code from PVE::API2::Storage::Content
For better code reuse.
2015-10-01 06:50:19 +02:00
Dietmar Maurer
746e530f72 white space cleanups: correctly intend code 2015-10-01 06:28:22 +02:00
Dietmar Maurer
180c8b020c complete_storage: correctly pass storage config to storage_ids() 2015-10-01 06:26:35 +02:00
Alen Grizonic
ba0cf557f4 pvesm list fix 2015-09-29 17:25:05 +02:00
Wolfgang Bumiller
399581a2b3 volume_snapshot_delete: deactivate before deleting 2015-09-25 13:41:04 +02:00
Dietmar Maurer
82548118ec drbd: use correct unit for storage sizes 2015-09-25 13:34:55 +02:00
Thomas Lamprecht
2e346fd478 drbd: return correct used/avail/total sizes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2015-09-25 13:10:05 +02:00
Wolfgang Link
02e797b8e1 extend functionality to (de)activate_volumes with snapshots
this extension provide the capability to activate or deactivate snapshot,
so we can use this e.g. for LXC backup in snapshot mode.
2015-09-22 12:18:09 +02:00
Dietmar Maurer
f7621c0126 improve bash completions 2015-09-18 09:33:09 +02:00
Dietmar Maurer
c669f42d4e convert pvesm into a PVE::CLI class 2015-09-18 08:22:04 +02:00
Wolfgang Link
c8943a85c9 remove non used parameter exclusive
This makes no sense because it should always be exclusive.
Also RDB checks it self.
LVM has not possibility to use lvchange.
DRBD is this feature not implemented.
2015-09-17 12:37:27 +02:00
Dietmar Maurer
097a2b2fcf fix lvm storage status 2015-09-11 12:13:00 +02:00
Dietmar Maurer
4f6a99d8c7 rbd: make activate_volume idempotent 2015-09-10 11:12:09 +02:00
Wolfgang Link
b50812f941 rbd: skip unmap if device is not mapped 2015-09-10 10:41:54 +02:00
Dietmar Maurer
c2c312175b drbd: also allow to configure content property 2015-09-10 08:41:00 +02:00
Dietmar Maurer
3c05693484 allow to create containers on DRBD (content == rootdir) 2015-09-10 08:29:23 +02:00
Wolfgang Link
515ef80bf5 improve free_image with deactivate_volumes
this is necessary to prevent an error when trying to remove a mapped volume.
2015-09-09 07:09:37 +02:00
Dietmar Maurer
84c8e52da9 use read_proc_mounts from PVE::ProcFSTools 2015-09-03 10:16:38 +02:00
Wolfgang Link
bb5520eb17 move read_proc_mounts from NFSPlugin to Tools.
this function will be uesd in more path of the code.
2015-09-03 10:12:07 +02:00
Wolfgang Bumiller
be48449c28 GlusterfsPlugin: replace Net::Ping with tcp_ping
Using PVE::Network::tcp_ping on port 7 (echo) for ipv6
support.
2015-08-31 15:50:56 +02:00
Wolfgang Bumiller
1689e627a6 iSCSI ipv6 support 2015-08-31 15:50:34 +02:00
Wolfgang Link
9660f71602 fix to prevent uninitialized variable massage. 2015-08-28 11:24:49 +02:00
Dietmar Maurer
48e27f7937 scan_zfs: ignore subvolumes generated by our ZFSPoolPlugin 2015-08-28 11:01:41 +02:00
Wolfgang Bumiller
3932390b11 allow subvolume selection in ZPool storage 2015-08-28 10:56:55 +02:00
Dietmar Maurer
e67069eb58 path: corretly implement path to snapshots 2015-08-27 12:03:46 +02:00
Wolfgang Link
1f79bb07f1 Add content type rootfs to RBD and extend manual 2015-08-26 15:39:09 +02:00
Alen Grizonic
2a2cf20aa9 remove storage access permissions after delete 2015-08-19 15:17:38 +02:00
Alen Grizonic
402df80b12 added remove storage (exist) check 2015-08-19 15:14:29 +02:00
Wolfgang Bumiller
53ec90e23e upload API: safer filename handling
Replace possibly-dangerous characters in uploaded filenames
with underscores, this includes spaces, colons, commas,
equal signs and any byte >= 128. Previously only spaces were
turned into underscores.

Also shell_quote the destination for scp.

Use '--' for some shell commands for safety.

Use brackets around the scp destination for ipv6 support.
2015-08-18 16:23:53 +02:00
Dietmar Maurer
a6f1262677 parse_volname: document return values 2015-08-12 07:34:58 +02:00
Dietmar Maurer
7800e84d28 parse_volname: always return image format 2015-08-12 07:01:07 +02:00
Dietmar Maurer
55525ad265 ZFSPoolPlugin: improve subvol support
- return format with parse_volname
- consider subvol in zfs_find_free_diskname
2015-08-12 06:49:15 +02:00
Dietmar Maurer
35533c68fb add subvol support for directory storage
This storage type does not support subvol quotas, so we
only allow this if vdisk_alloc is called with size=0.
2015-08-12 06:11:10 +02:00
Dietmar Maurer
dec97937f2 zfs_parse_zvol_list: do not add pool to volname
Else, alloc and list return different volids.
2015-08-10 16:53:19 +02:00
Dietmar Maurer
d73060becd storage_info: fix content filter
List all storages if $content filter is undefined.
2015-08-10 10:05:01 +02:00
Alexandre Derumier
9f20a8a643 rbd: add support to krbd
Can be use by lxc (but also qemu)

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
2015-08-07 11:56:40 +02:00
Dietmar Maurer
583c2802f8 storage status API: allow to filter multiple content types 2015-08-07 09:46:49 +02:00
Wolfgang Link
27ff0e99e2 fix bug #664
Catch the error, if the dataset doesn't exists.
If it will not catched, you can't remove a VM.
2015-07-30 11:50:46 +02:00
Wolfgang Bumiller
afdfbe5594 mask world r and group wx permissions in vdisk_alloc
Avoid world-readable disk files being created as suggested
in #416 by setting an umask to strip world permissions as
well as group write/exec permissions before calling
alloc_image.
2015-07-30 08:49:46 +02:00
Dietmar Maurer
13d2cb7980 allow templates using .xz compression 2015-07-01 10:06:24 +02:00
Wolfgang Link
d6d924d0bf Fix ZFSPoolPlugin path for subvol
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-06-22 11:03:11 +02:00
Dietmar Maurer
30a1369b91 drbd: improve error messages 2015-06-19 12:10:45 +02:00
Dietmar Maurer
ae9e512e91 drbd: implement deactivate_volume (but code is disabled for now) 2015-06-18 13:06:30 +02:00
Dietmar Maurer
9913665327 drbd: use old code to wait for diskless assignment 2015-06-18 12:21:44 +02:00
Dietmar Maurer
b0e0ed1abe drbd: remov hack, because bug is fixed upstream
see http://git.drbd.org/gitweb.cgi?p=drbdmanage.git;a=commitdiff;h=8056e58b929673afb2d4a33d82fdeea4d43f8667
2015-06-18 11:55:56 +02:00
Dietmar Maurer
31ba75ff58 drbd: correctly check all result codes 2015-06-18 11:45:04 +02:00
Dietmar Maurer
4959ea20c1 drbd: correctly wait for diskless assignments 2015-06-18 10:43:43 +02:00
Dietmar Maurer
d1e7ac11de drbd: improve workaround for set_drbdsetup_props 2015-06-18 06:21:38 +02:00
Dietmar Maurer
98e250aa41 drbd: call set_drbdsetup_props after auto-deploy
Else it gets not assigned. For some unknown reason, we need to sleep
to avoid a split brain?!
2015-06-17 11:33:49 +02:00
Dietmar Maurer
07af07cc16 drbd: set allow-two-primaries=yes
So that we can do live migrations.
2015-06-17 08:57:32 +02:00
Wolfgang Link
ee302b1caa Fix nfsscan
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-06-10 10:55:06 +02:00
Wolfgang Bumiller
6564419625 include ::1 in localhost check in get_active_server 2015-05-27 08:29:39 +02:00
Wolfgang Bumiller
f8b0d82fe6 NFS: replace rpcinfo with showmounts
rpcinfo from rpcbind-0.2.1 in debian doesn't support ipv6 addresses.
At the same time the used command only actually tests for
portmapper/rpcbind availability, not for NFS directly.
Storage::scan_nfs uses /sbin/showmount to get a list of NFS exports from a
server and happily accepts ipv6 addresses. It is also more specific to NFS.

Replacing the rpcinfo call with showmount here means checking explicitly
for NFS and supporting IPv6 without the need for an updated rpcbind
package.
2015-05-21 11:49:35 +02:00
Wolfgang Bumiller
da63f58832 NFSPlugin: deal with ipv6 addresses and fix regexps
NFS needs brackets around ipv6 addresses.
Also: nfs_is_mounted needs to quote the variables. This becomes apparent
when ipv6 addresses are used as then the address would otherwise be
treated as a character class, causing the check to always fail.
2015-05-21 11:49:21 +02:00
Wolfgang Bumiller
6bf617a954 allow ipv6 addresses in Storage::verify_server
Replaced regex with existing PVE::JSONSchema::verify_... functions.
2015-05-21 11:48:08 +02:00
Dietmar Maurer
69a093c7fd drbd: avoid warning about undefined value 2015-05-18 14:15:35 +02:00
Dietmar Maurer
47dbb901c3 fix typo 2015-05-18 14:12:59 +02:00
Wolfgang Bumiller
c67daeac17 ipv6 support for Storage::resolv_server
While in posix gethostbyname(3) does support ipv6, perl's gethostbyname
usually returns wrong results for names, or no results for ipv6
addresses. Since we provide a getaddrinfo helper already, we now use
that instead.
2015-05-12 10:39:15 +02:00
Wolfgang Link
f5640e7d3b remove running from Storage and check it in QemuServer
It is better to check if a VM is running in QemuServer then in Storage.
for the Storage there is no difference if it is running or not.

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-05-06 11:42:38 +02:00
Dietmar Maurer
1ccae4490e zfspool: new format 'subvol' to handle filesystems inside datasets
We use this for containers.
2015-05-01 14:46:17 +02:00
Dietmar Maurer
4cb6e060c3 correctly parse lxc backup files 2015-05-01 11:29:15 +02:00
Dietmar Maurer
93124ef4fe ZFSPoolPlugin: allow nested datasets 2015-04-30 06:48:39 +02:00
Dietmar Maurer
6d788031b0 add newlines to error messages 2015-04-29 06:59:48 +02:00
Dietmar Maurer
e0852ba7dc whitespace cleanups 2015-04-27 10:44:32 +02:00
Wolfgang Link
3d621977f4 implement offline migration on zfs
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-04-27 10:44:23 +02:00
Dietmar Maurer
466183d677 DRBDPlugin: hack - wait until device is acessible 2015-03-23 12:50:13 +01:00
Dietmar Maurer
28d5851293 DRBDPlugin: automatically create diskless assignments 2015-03-23 11:51:02 +01:00
Dietmar Maurer
fb0e1d93a7 DRBDPlugin: correctly use default redundancy 2015-03-23 09:38:39 +01:00
Dietmar Maurer
d26e1891ff mark drbd as shared storage 2015-03-21 11:20:18 +01:00
Dietmar Maurer
5d6a88b0c4 DRBDPlugin: fix bug in status 2015-03-21 11:02:47 +01:00
Dietmar Maurer
eab90afd46 DRBDPlugin: use Net::DBus 2015-03-21 10:58:06 +01:00
Dietmar Maurer
14770890e2 add dummy for DRBD9 2015-03-20 17:44:15 +01:00
Wolfgang Link
37121146ec fix rpcinfo path
change path in jessie of package rpcbind
from /usr/bin/rpcinfo to /usr/sbin/rpcinfo

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-03-06 10:33:26 +01:00
Wolfgang Link
1f390a3030 fix bug #602: now zfs will wait 5 sec if error msg is "dataset is busy"
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-03-06 10:15:40 +01:00
Pawel Boguslawski
399ab2b605 Fixed LV cleaning function. saferemove_throughput parameter added.
Signed-off-by: Pawel Boguslawski <pawel.boguslawski@ib.pl>
2015-03-06 10:12:52 +01:00
Alexandre Derumier
a8176f5458 rbd: monhost string : escape only ":" character
we need to escape ":" used to defined mon ports

"10.5.0.11:6789; 10.5.0.12:6789; 10.5.0.13:6789"

->

"10.5.0.11\:6789; 10.5.0.12\:6789; 10.5.0.13\:6789"

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
2015-03-06 10:10:10 +01:00
Wolfgang Link
82e08809fd change var name to prevent side effects.
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-02-16 11:35:13 +01:00
Wolfgang Link
4eff810db8 Bug-fix alloc_image in ZFSPlugin.pm
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-02-16 11:31:28 +01:00
Wolfgang Link
8a55ff7d39 bug-fix for size output
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-02-12 11:47:45 +01:00
Dietmar Maurer
76fd7dc7c2 ZFSPoolPlugin: use udevadm settle with --exit-if-exists
And silently ignore error if that command fails.
2015-02-12 11:47:45 +01:00
Dietmar Maurer
f824c72250 cleanup error messages 2015-02-12 09:40:17 +01:00
Dietmar Maurer
b3f302c638 call volume_rollback_is_possible from PVE::Storage::volume_snapshot_rollback
That way it is automatically called for all plungins.
2015-02-12 09:28:52 +01:00
Wolfgang Link
1597f1f9ad Storage: add method volume_rollback_is_possible
add method volume_rollback_is_possible and redactor
Improve error handling
If snapshot is not reversible catch it before vm will lock and shutdown.
This is the case if zfs has an younger snapshot.

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-02-12 09:15:02 +01:00
Wolfgang Link
a4034b9f19 zfspoolplugin: fix volume_resize
copy methode volume_resize form ZFSPlugin.pm to ZFSPoolPlugin.pm
refactor volume_resize

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-02-10 12:48:34 +01:00
Wolfgang Link
86d472394a zfs: auto import after reboot
this is necessary, because after a reboot all pools except rpool are gone

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-02-09 12:26:28 +01:00
Wolfgang Link
90a11abe78 ZfsPoolPlugin: fix add disks
now it is possible add disk to an vm on rpool

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-02-09 12:09:43 +01:00
Wolfgang Link
9b61bc9e88 zfs: fix wait by alloc_image
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-01-30 06:24:56 +01:00
Dietmar Maurer
584d97f646 implement zfsscan to list local zfs pools 2015-01-26 09:05:09 +01:00
Dietmar Maurer
0a3d992f72 move activate_storage into ZFSPoolPlugin 2015-01-24 14:30:10 +01:00
Dietmar Maurer
85fda4dd7f rename ZFSDirPlugin to ZFSPoolPlugin
Turned out it makes no sense to duplicated DirPlugin features. So I
also changed the name to make it less confusing. So we can only
create zvols inside a zfs pool with this plugin.
2015-01-24 14:17:55 +01:00
Dietmar Maurer
0a71b39457 zfs: remove useless warning inside clone_image 2015-01-23 12:57:38 +01:00
Wolfgang Link
d3a282e83e zfs: move and refactor code
copy and modify create_base and refactor clone_image

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-01-23 12:57:38 +01:00
Dietmar Maurer
2fc59177c9 zfs: implement zfs_get_latest_snapshot
To improve code sharing.
2015-01-23 12:57:38 +01:00
Wolfgang Link
2b40ffaea9 zfs: move code fix metode call
duplicate code because can't refactor volume_snapshot_rollback

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-01-23 12:57:38 +01:00
Wolfgang Link
d4c63dc147 zfs: move code
move activate_volume deactivate_volume from ZFSPlugin to ZFSDirPlugin

Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-01-23 12:57:38 +01:00
Wolfgang Link
f3e632d073 zfs: ZFSDirPlugin add methode path
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
2015-01-23 12:57:38 +01:00