Commit Graph

1961 Commits

Author SHA1 Message Date
Fabian Grünbichler
c1ec1acbde file-restore: pass in volume ID or name
instead of just the snapshot for consistency with other API endpoints,
and possible future extension to VMA backups (where 'snapshot' would be
a rather strange terminology).

add some additional checks (pbs storage type, backup volume type),
completion and magic (allow passing in either a full volume ID with
correct storage, or just the volume name, or just the snapshot for
easier API/CLI usage/convenience).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-04-23 14:09:54 +02:00
Fabian Grünbichler
82f764e119 file-restore: return perl-y booleans
like we do in most of our API.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-04-23 14:09:54 +02:00
Stefan Reiter
f1a3ce3b17 add FileRestore API for PBS
Includes list and restore calls.

Requires VM.Backup and Datastore.Audit permissions, for the accessed
VM/CT and containing datastore respectively.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-23 14:09:48 +02:00
Fabian Grünbichler
6035a5dfb1 api: fix typo in error message
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-04-23 12:29:36 +02:00
Fabian Grünbichler
c56f7a71af pbs: allow setting up a master key
similar to the existing encryption key handling, but without
auto-generation since we only have the public part here.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-04-22 21:56:31 +02:00
Fabian Ebner
3c93115570 rbd: fix typo in error message
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-20 18:21:31 +02:00
Fabian Ebner
ceb7b1ed09 diskmanage: get_partnum: fix check
Not replacing it with return, because the current behavior is dying:
    Can't "next" outside a loop block
and the single existing caller in pve-manager's API2/Ceph/OSD.pm does not check
the return value.

Also check for $st, which can be undefined in case a non-existing path was
provided. This also led to dying previously:
    Can't call method "mode" on an undefined value

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-20 18:13:18 +02:00
Fabian Ebner
415dc3985d diskmanage: improve setting usage for whole disk with include-partitions
in case a disk with partitions also has an fstype set, which happens for our ZFS
boot disks. Do not change the behavior without include-partitons, as we
prefer(red) to be more specific than simply 'partitions' then.

Reported in the enterprise support channel.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-16 12:52:08 +02:00
Fabian Grünbichler
1ebd925dcf import: allow import from UNIX socket
this allows forwarding over websockets without requiring a (free) port.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-04-16 12:23:43 +02:00
Thomas Lamprecht
9e47ad47ec bump version to 6.3-9
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 16:14:36 +02:00
Thomas Lamprecht
0d8ed0a313 test: rbd: improve usage output
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 15:43:42 +02:00
Thomas Lamprecht
23ba4ac2c6 test: rbd: typo fix
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 15:27:33 +02:00
Thomas Lamprecht
33191d3266 test: rbd: indendation cleanup, allow -d short-opt for debug
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 15:27:26 +02:00
Thomas Lamprecht
0d7c9ca188 test: rbd: actually use debug helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 15:27:23 +02:00
Fabian Ebner
bef7920d1e volume export/import: allow uppercase letters
Bug reported in the community forum[0].

Currently, it's possible to break replication by:
1. have an existing snapshot whose name contains an uppercase letter
2. set up a replication job and run it
3. rollback to the existing snapshot
4. replicate again -> fails

The failure occurs, because after step 3, the most recent common snapshot is the
previously existing one and currently no uppercase letters are allowed for
export/import.

The pve-snapshot-name option uses the CONFIGID_RE
    qr/[a-z][a-z0-9_-]+/i
so it cannot be used here, because it would not allow for e.g. '__migrate__'.
Simply allow uppercase letters, to be backwards compatible and allow all
possible pve-snapshot-name values.

There is still an issue if there also was state volume, but that's a different
bug[1].

[0]: https://forum.proxmox.com/threads/solved-migration-error-base-value-does-not-match-the-regex-pattern.85946/
[1]: https://bugzilla.proxmox.com/show_bug.cgi?id=3111

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-12 14:52:29 +02:00
Thomas Lamprecht
9407a7a338 tests: rbd ns: exit cleanly on help, and allow short-opt -h
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 14:46:12 +02:00
Thomas Lamprecht
876d268298 tests: rbd ns: avoid underscore in CLI param name
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 14:42:38 +02:00
Aaron Lauterer
8c858f7eeb fix #3345: zfs: restore container volume to ZFS with size 0
A restore to ZFS for a container which has a volume (rootfs / mount
point) of size 0 failed because the refquota property does not accept
'0k' but wants 'none' in that situation.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-12 14:37:50 +02:00
Thomas Lamprecht
c27fe64810 rbd: make volume param for get_rbd_path to allow further use
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 14:19:48 +02:00
Thomas Lamprecht
ed7ea5a352 rbd: list images: early return to avoid indentation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 13:51:15 +02:00
Thomas Lamprecht
a3cad0b50d rbd: list images: sort by keys when pushing on result array
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 13:49:56 +02:00
Thomas Lamprecht
6d0d0a977d rbd: indentation and whitespace cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 13:48:27 +02:00
Thomas Lamprecht
22265bd990 rbd: get kernel device sub returns a path, not a name
also transform to private sub instead of local variable closure.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 13:45:21 +02:00
Thomas Lamprecht
72bbd8a6f7 rbd: consistent closure call style
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 13:43:33 +02:00
Thomas Lamprecht
92a7826f88 rbd: build cmd: allow "falsy" namespace value also here
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 13:43:33 +02:00
Thomas Lamprecht
4cf696f66e rbd: use private sub for get_rbd_path
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 13:43:33 +02:00
Aaron Lauterer
9f3d6f3bb9 rbd: add integration test for namespace handling
This test is intended to be run on a hyperconverged PVE cluster to test
the most common operations of VMs using a namespaced Ceph RBD pool.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2021-04-09 12:56:21 +02:00
Aaron Lauterer
e9bc993138 rbd: fix #3286 add namespace support
This patch introduces support for Cephs RBD namespaces.

A new storage config parameter 'namespace' defines the namespace to be
used for the RBD storage.

The namespace must already exist in the Ceph cluster as it is not
automatically created.

The main intention is to use this for external Ceph clusters. With
namespaces, each PVE cluster can get its own namespace and will not
conflict with other PVE clusters.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2021-04-09 12:56:21 +02:00
Aaron Lauterer
aeb007cb76 rbd: centralize rbd path concatenation
The <pool>/<image> paths are needed in quite a lot of places. Having one
single place where they are created helps to reduce duplicate code and
makes it easier to introduce new features.

The 'add_pool_to_disk' sub was already doing that but the name was not
really fitting. This commit renames it to the more general
'get_rbd_path' and changes the second parameter to the more widely used
$volume instead of $disk.

Furthermore, all occurences where "$pool/$volume" has been concatenated
have been replaced with a call to get_rbd_path.

Plus some minor code style cleanups for long function calls that were
touched.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2021-04-09 12:56:21 +02:00
Thomas Lamprecht
cba2b7c1d4 prune backups: improve internal errors messages slightly
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 12:49:55 +02:00
Thomas Lamprecht
4b702ac361 plugin: get_subdir_files: add comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-09 12:46:46 +02:00
Fabian Ebner
68ce0b31e8 prune backups: make vmid filtering more robust
by relying on archive_info's vmid first. archive_info is already used to
determine if it's a standard name, and in that case the vmid is certainly set.

Also add asserts to make sure we got what we expected.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-09 12:41:34 +02:00
Fabian Ebner
edb92f707a plugin: subdir files: backup: don't match for vmid against the full path
Only match against the file name to avoid false positives with
directory names containing "-$vmid-".

Found while trying to debug/reproduce a forum thread[0], but the path
there should not be affected by this...

[0]: https://forum.proxmox.com/threads/vzdump-removing-too-many-backups.87072/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-09 12:41:10 +02:00
Thomas Lamprecht
c241c2159d bump version to 6.3-8
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-01 15:04:16 +02:00
Dominic Jäger
4b84ad5e25 Optionally allow blockdev in abs_filesystem_path
This is required to import from LVM storages, for example

Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
2021-03-31 11:00:25 +02:00
Fabian Ebner
342a56805c fix #3348: NFS: select correct transport to check for service
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-31 10:22:52 +02:00
Fabian Grünbichler
dfa374d320 fix #3363: avoid undef-warning for PBS crypt-mode
it is optional after all, and missing (/None) for files stored in the
snapshot dir but not referenced in the manifest for whatever reason.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-03-31 10:22:52 +02:00
Fabian Ebner
2c5246e1ea vdisk list: allow specifying content type
and only scan storages that support it if specified.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-31 10:22:52 +02:00
Fabian Ebner
26d7314be0 Revert "vdisk list: only collect images from storages with an appropriate content type"
This reverts commit a44c18925d and adds a reminder
comment.

The mentioned commit is actually a backwards-incompatible change that leads to
slightly different behavior when migrating a VM with volumes on a misconfigured
storage. For example, unreferenced volumes on a misconfigured storage won't be
picked up, even though they were before. And for referenced volumes on a
misconfigured storage, the disk size would not be updated on migration anymore.

We should wait until the next major release for this change and then also
re-evaluate the migration behavior with misconfigured disks.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-31 10:22:52 +02:00
Dylan Whyte
2bce96c513 fix #3354: support notes on ceph backups
use DirPlugin's get/update_volume_notes implementation (which all the
other supported file systems use)

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2021-03-31 10:22:52 +02:00
Fabian Ebner
061c7e5e3d tests: zfs: complain when a first sub-test dies
Previously, $count was not increased and no test failure was reported.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-15 13:36:53 +01:00
Fabian Ebner
a44c18925d vdisk list: only collect images from storages with an appropriate content type
Only these storages are activated in the first place, and it's bad behavior to
list images when no appropriate content type is not set.

For example, on VM destruction, this avoids unreferenced images to be deleted
from a storage with only 'backup' content type set, which is supposedly what
happened in this[0] forum thread.

(Some) callers expect all keys to be present and valid array references in the
result, so initialization is needed.

Now, the enabled check is already done by the preceding code for every element
that is iterated over, and thus isn't needed in the main loop anymore.

[0]: https://forum.proxmox.com/threads/erasing-all-vm-disks-after-a-failed-vm-migration-task.85068

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-15 13:36:53 +01:00
Fabian Ebner
1d6c548889 api: disk list: allow if an audit permission for the node is present
as that seems to be the more natural permission path for listing a nodes local
disks. For backwards compatibility, the old permission check has to be kept
(relevant with propagate=0).

This API call was originally part of the Ceph API and got copied here later,
which might explain the current permission check.

In the UI, the Disk panel is visible with a node audit permission, but the API
call itself failed without the '/' audit permission.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-11 14:16:21 +01:00
Thomas Lamprecht
6c460330bb d/control: record pve-qemu-kvm | qemu-utils build depenency for tests
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-02 17:20:57 +01:00
Stoiko Ivanov
977b80c8ab disks: zfs: scan is only returned optionally
the line is not present if a zpool has never been scrubbed before
(e.g. if it's freshly setup)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2021-02-26 09:10:08 +01:00
Thomas Lamprecht
c726ce1705 bump version to 6.3-7
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-02-19 15:49:27 +01:00
Fabian Ebner
7cdc75a290 storage migration: insecure: improve logging
by including the message/error from the remote side. Some people on the forum[0]
ran into 'no tunnel IP received', but without information from the remote side
it's hard to tell why.

[0]: https://forum.proxmox.com/threads/failed-no-tunnel-ip-received.80172

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-19 15:29:04 +01:00
Fabian Ebner
525b4a6ebe Diskmanage: replace check for zpool binary with a function and mock it
so the test still works when it's not installed.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-19 15:29:04 +01:00
Thomas Lamprecht
26de022b56 zpool: activate: move mount check out and make program flow easier
Early return when mounted heuristics returns true, that allows to get
rid of an indentation level.

Moving the heuristic out makes the activate method smaller and easier
to grasp

Best viewed with ignoring whitespace changes (`git show -w`).

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-02-19 15:21:21 +01:00
Thomas Lamprecht
9440330aba zpool: activate: don't eval procfs read, if it fails it should be fatal
highly unlikely to fail in our setups, most realistic case is when
procfs is not mounted at /proc, which breaks much else anyway and is
a requirement

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-02-19 15:06:22 +01:00