Commit Graph

2945 Commits

Author SHA1 Message Date
Thomas Lamprecht
c09667bc9d bump version to 7.0-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-13 19:11:22 +02:00
Thomas Lamprecht
b818a0504c debian: update compat, copyright, ..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-13 19:09:29 +02:00
Thomas Lamprecht
c948e143d8 bump version to 6.4-2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-29 16:16:51 +02:00
Fabian Grünbichler
3dc33a728a fix #2862: allow sata/ide template backups
for IDE and SATA, setting the whole drive into readonly mode is not
possible. skip the readonly flag for such drives as a workaround until
we find a better solution.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-04-29 16:15:53 +02:00
Dominik Csapak
30664f14ff fix bootdisk_size for new bootorder config scheme
Previously, we ever only had a single boot *disk*, while possibly
having multiple cdroms/nics in the boot order

e.g. the config:

 boot: dnc
 bootdisk: scsi0
 ide0: media=cdrom,none
 scsi0: xxx
 net0: ...

would return the size of scsi0 even though it would first boot
from cdrom/network.

When editing the bootorder with such a legacy config, we
remove the 'bootdisk' property and replace the legacy notation
with an explicit order, but we only search the first disk
for the size now.

Restore that behaviour by iterating over all disks in the boot
order property string until we get one that is not a cdrom
and has a size.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-29 16:15:33 +02:00
Thomas Lamprecht
8f43ac4893 Revert "migration: do not set default speed limit"
The default was changed for 5.2, so while it is not 32 MiB/s anymore,
it is still 128 MiB/s which I did not notice on my 1 Gbps (or < 125
MiB/s) setup. For users with links faster than one gigabit it now did
some limiting - so setup a very high limit so than even 100G should
not max this out.

This reverts commit a89bd10084.
2021-04-29 15:48:21 +02:00
Thomas Lamprecht
ba2a06fda2 bump version to 6.4-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-23 16:26:59 +02:00
Fabian Ebner
9938d24df2 migrate: fix memory migration start time
The variable is only ever used for calculating the average speed of memory
migration, but it was set before disk mirroring already. But the disk
sizes are not included in the calculation, resulting in (very) wrong values.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-23 15:00:44 +02:00
Fabian Ebner
6629f976ac qemu_img_convert: add missing newline for progress output
which was accidentally removed by b5e9d97bdf.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-22 11:37:00 +02:00
Dylan Whyte
ebce523987 fix #3369: auto-start vm after failed stopmode backup
Fixes an issue in which a VM/CT fails to automatically restart after a
failed stop-mode backup.

Also fixes a minor typo in a comment

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-04-22 11:35:20 +02:00
Stefan Reiter
f755117071 live-restore: hold 'create' lock during operation
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:46:05 +02:00
Stefan Reiter
fefd65a1d9 live-restore: don't remove VM on error
Potentially an admin can still recover some data, or wants to inspect
the state.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:46:05 +02:00
Stefan Reiter
020dd358f9 qmrestore: add live-restore option
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:46:05 +02:00
Fabian Ebner
ab5b97d8a8 drive: volume in-use check: remove unused closure parameter
and simplify the calling iteration.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-19 22:09:17 +02:00
Thomas Lamprecht
b68a957b2e migration: keep log rate steady if polling gets more frequent
Either we're done in a few seconds anyway, or if the VM dirties lots
of pages we need quite a bit of time, and then it does not help to
output roughly the same status 10 times a second...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-19 22:08:19 +02:00
Thomas Lamprecht
0fca250af0 migration: rework logging to more humand friendly, less spammy
* use render_bytes where possible, to get quick to read and grasp
  units printed
* xbzrle is only interesting if actually pages/bytes are send using
  it, so only log in that case
* log if VM dirties more than we send
* log current speed we get from QEMU

In general there are less lines logged and huge integers are avoided.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-19 21:54:37 +02:00
Thomas Lamprecht
e693c49190 migration: factor out variable + code cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-19 21:51:21 +02:00
Thomas Lamprecht
7de328c629 migration: log: s/migration_caps/migration capabilities/
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-19 21:48:31 +02:00
Thomas Lamprecht
a89bd10084 migration: do not set default speed limit
the claim that QEMU limits this to 32M otherwise is bogus, at least
with any current QEMU version..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-19 21:46:52 +02:00
Thomas Lamprecht
6539865a9d migration: refactor and tidy-up code
Use an early die so that the rest can loose an indentation level for
the actual migration status reporting code

Extract common used members of the stat hash for shorter code.

use `git show -w --word-diff=color --word-diff-regex='\w+'` for
getting a better view of actual changes

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-19 14:59:54 +02:00
Fabian Ebner
0783c3c271 migration: move finishing block jobs to phase2 for better/uniform error handling
avoids the possibility to die during phase3_cleanup and instead of needing to
duplicate the cleanup ourselves, benefit from phase2_cleanup doing so.

The duplicate cleanup was also very incomplete: it didn't stop the remote kvm
process (leading to 'VM already running' when trying to migrate again
afterwards), but it removed its disks, and it didn't unlock the config, didn't
close the tunnel and didn't cancel the block-dirty bitmaps.

Since migrate_cancel should do nothing after the (non-storage) migrate process
has completed, even that cleanup step is fine here.

Since phase3 is empty at the moment, the order of operations is still the same.

Also add a test, that would complain about finish_tunnel not being called before
this patch. That test also checks that local disks are not already removed
before finishing the block jobs.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
a6be63ac9b migration: split out replication from scan_local_volumes
and avoid one loop over the config, by extending foreach_volid to include the
drivename.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
4b26ffbfa5 migration: keep track of replicated volumes via local_volumes
by extending filter_local_volumes.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
efe0d457c6 migration: use storage_migration for checks instead of online_local_volumes
Like this we don't need to worry about auto-vivifaction.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
eb5751ba02 migration: cleanup_remotedisks: simplify and include more disks
Namely, those migrated with storage_migrate by using the information from
volume_map. Call cleanup_remotedisks in phase1_cleanup as well, because that's
where we end if sync_offline_local_volumes fails, and some disks might already
have been transfered successfully. Note that the local disks are still here, so
this is fine.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
ad8b9d5e2d migration: simplify removal of local volumes and get rid of self->{volumes}
This also changes the behavior to remove the local copies of offline migrated
volumes only after the migration has finished successfully (this is relevant
for mixed settings, e.g. online migration with unused/vmstate disks).

local_volumes contains both, the volumes previously in $self->{volumes}
and the volumes in $self->{online_local_volumes}, and hence is the place
to look for which volumes we need to remove. Of course, replicated
volumes still need to be skipped.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
efbbe59da4 migration: add nbd migrated volumes to volume_map earlier
and avoid a little bit of duplication by creating a helper

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
c3417e3b6e migration: save targetstorage and bwlimit in local_volumes hash and re-use information
It is enough to call get_bandwith_limit once for each source_storage.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
2c4ba4c3ee migration: fix calculation of bandwith limit for non-disk migration
The case with:
1. no generic 'migration' limit from the storage plugin
2. a migrate_speed limit in the VM config
was broken. It would assign 0 to migrate_speed when picking the minimum value
and then default to the default value. Fix it by checking if bwlimit is 0
before picking the minimum.

Also, make it a bit more readable by avoiding the trick of //-assigning bwlimit
before the units match up and relying on getting back the original bwlimit value
as the minimum. Instead, only ||-assign after the units match up and don't rely
on other things.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
3276a43470 migration: split out config_update_local_disksizes from scan_local_volumes
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
62a4c963b8 migration: avoid re-scanning all volumes
by using the information obtained in the first scan. This
also makes sure we only scan local storages.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
d10b78f4d2 migration: split sync_disks into two functions
by making local_volumes class-accessible. One functions is for scanning all local
volumes and one is for actually syncing offline volumes via storage_migrate. The
exception is replicated volumes, this still happens during the scan for now.

Also introduce a filter_local_volumes helper, to makes life easier.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
93a1c63f4c test: migration: add parse_volume_id calls
so it fails when something bad comes in.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:30:41 +02:00
Fabian Ebner
eabac302ba restore: update config: remove unused parameter
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:10:28 +02:00
Fabian Ebner
c62d7cf547 test: add tests for restoring config
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:10:28 +02:00
Thomas Lamprecht
41ab78c626 d/control: bump pve-storage dependency for new vdisk param
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-18 18:04:29 +02:00
Fabian Ebner
d0ff75d9b4 filter by content type when using vdisk_list
except for migration, where it would be subtly backwards-incompatible. Since
there is a scan_volids call for migration, we can't default to filtering in
scan_volids just yet.

Also allows to get rid of the existing filtering hack in rescan().

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 18:03:44 +02:00
Stefan Reiter
d4be7f31b5 cfg2cmd: fix +pveN machine types with pxe
Pinned machine versions like "pc-i440fx-4.2+pve2.pxe" would otherwise
get a second "+pve0" suffix, which is incorrect.

Also deal with non-pve pinned versions correctly, i.e.
"pc-i440fx-5.2.pxe" becomes "pc-i440fx-5.2+pve0.pxe".

Handle .pxe suffixes in Machine.pm as well, and add two test cases.

Co-developed-by: Luca Berneking <luca@berneking.net>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-18 17:58:56 +02:00
Fabian Ebner
a0d8d8592a drive: volume in-use check: fix fallback path comparison
When checking whether a volume is still referenced by a snapshot, the volid
itself is first checked. When the volid is different, we fall back to comparing
the path.

As the first value to be compared is a volume's path, the second value better be
a volume's path too, and not a snapshot's path.

See also 77019edfe0 for historical context.

The error that led me here:
* had a VM with ZFS over iSCSI storage with an exsiting snapshot
* add new unused drive
* try to remove the unsued drive
* fails, because ZFS (not Pool!) Plugin does not support snapshot paths.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-04-18 17:26:05 +02:00
Thomas Lamprecht
04fac98acc bump version to 6.3-11
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 18:32:10 +02:00
Thomas Lamprecht
67daf6921b drive mirror: stop logging progress for a disk after it got ready
If, why ever, got "not-ready" again we'd log again the next round.

Improves the behavior for multiple disks, especially on migration
where we mirrored the local disks one by one, but kept reporting on
prev. ones.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 17:52:54 +02:00
Thomas Lamprecht
b5e9d97bdf image convert: use human-readable units in progress report
similar to what driver mirror monitor was changed too

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 17:51:04 +02:00
Thomas Lamprecht
fd70c84362 indentation line-length cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 17:50:13 +02:00
Thomas Lamprecht
bb419195f9 restore PBS: use actual PVE::QemuConfig interface for destroying a config on error
avoid further spaghettification of our code base...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-06 19:43:47 +02:00
Thomas Lamprecht
bfb1267858 pbs_live_restore: code cleanup, avoid prefixin local package
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-06 19:43:03 +02:00
Thomas Lamprecht
3b56383bdb mirror monitor: rework periodic status reporting
orient on the backup output which got reworked for PVE 6.2/6.3

Avoid overwhelming the user with redundant information, and use human
readable units.

before:
> restore-drive-scsi5: transferred: 167772160 bytes remaining: 8422162432 bytes total: 8589934592 bytes progression: 1.95 % busy: 1 ready: 0

after:
> restore-drive-scsi0: transferred 720.0 MiB of 32.0 GiB (2.20%) in 12s

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-06 19:40:21 +02:00
Thomas Lamprecht
a09b39f163 live restore: slightly more status output
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-06 19:40:08 +02:00
Thomas Lamprecht
1057fc7436 mirror monitor: avoid overlong hash access, use intermediate variable
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-06 17:46:19 +02:00
Thomas Lamprecht
0ea24bf080 mirror monitor: refactoring/code cleanup
mostly s/\$job/$job_id/ and s/foreach/for/ + sort.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-06 16:59:18 +02:00
Thomas Lamprecht
8986e36e85 live restore: start/delete blockdev jobs in deterministic order
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-06 16:51:04 +02:00