This fixes an issue when migrating a VM with an unused volume with format
qcow2 or vmdk. Since 'snapshots' wasn't set, storage_migrate wanted to
export/import with format raw+size instead. Therefore it used (instead of
just 'dd') 'qemu-img convert', which fails when its output leaves through
a pipe. Upon importing, a second error is present, because the format from
the volume ID doesn't match the format of the stream and there is no
conversion yet.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
LGTM-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
The initialization for the drive keys in $confdesc is changed
to be a single for-loop iterating over the keys of $drivedesc_hash and
the initialization of the unusedN keys is move to directly below it.
To avoid the need to change all the call sites, functions with more than
a few callers are exported from the submodule and imported into QemuServer.pm.
For callers of the now imported functions within QemuServer.pm, the prefix
PVE::QemuServer is dropped, because it is unnecessary and now even confusing.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
migrate_set_downtime, migrate_set_speed and migrate-set-cachesize have
all been deprecated since 2.8 or 2.11 [0]. They still work, but no
reason not to use the correct version.
Note that the downtime-limit parameter switched from seconds to
milliseconds, so convert to that. Slightly improve log output with units
while at it.
[0] https://qemu.weilnetz.de/doc/qemu-doc.html#Deprecated-features
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
QEMU usually only prints warnings and errors and stays silent otherwise,
so it makes sense to just log all of it's output.
Prefix it with '[<target_hostname>]' to indicate that the output is
coming from the remote node, so users know where to search for the
error.
Side effect is that the 'VM start' task created by the migration will
now show the "QEMU:" prefix, but it's still very readable IMHO.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
By default run_command prints the entire commandline executed when an
error occurs, but QEMU and our migrate command are not only
uninteresting to the user[*] but also annoyingly long. Hide them and only
print the exit code.
[*] Especially our migrate command, since it can't be manually executed
anyway. QEMU's commandline *might* contain something interesting, but is
so long that it's tricky to parse anyway, any a user can always call 'qm
showcmd --pretty'.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Split out 'update_disksize' from the renamed 'update_disk_config' to
allow code reuse in QemuMigrate.
Remove dots after messages to keep style consistent for migration log.
After updating in sync_disks (phase1) of migration, write out updated
config. This means that even if migration fails or is aborted in later
stages, we keep the fixed config - this is not an issue, as it would
have been fixed on the next attempt anyway, and it can't hurt to have
the correct size instead of a wrong one either way.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
...PVE::QemuServer::Machine.
qemu_machine_feature_enabled is exported since it has a *lot* of users
in PVE::QemuServer and a long enough name as it is.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
QMP and monitor helpers are moved from QemuServer.pm.
By using only vm_running_locally instead of check_running, a cyclic
dependency to QemuConfig is avoided. This also means that the $nocheck
parameter serves no more purpose, and has thus been removed along with
vm_mon_cmd_nocheck.
Care has been taken to avoid errors resulting from this, and
occasionally a manual check for a VM's existance inserted on the
callsite.
Methods have been renamed to avoid redundant naming:
* vm_qmp_command -> qmp_cmd
* vm_mon_cmd -> mon_cmd
* vm_human_monitor_command -> hmp_cmd
mon_cmd is exported since it has many users. This patch also changes all
non-package users of vm_qmp_command to use the mon_cmd helper. Includes
mocking for tests.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Live migration with a local cloudinit disk was never intended to work. It did
however work to an extent that the migration completed but the disk on the
source node could not be deleted. Now die if a live migration is started with
a local cloudinit disk.
With the GUI changes live migration is already disabled as it recognizes the
cloudinit disk as a local resource.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
for some reason not setting port results in a port of '65535' which
triggers an execption in http-server anyevent, so we set the port to 0
also, we have to read the ticket from stdin even for 'unix' type secure
migration
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
used for online local disks via qemu_drive_mirror
Add TODO comment for offline disks, as clone_disk calls `qemu-img
convert`, which does not have a bandwidth limit parameter.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
The 'migrate_speed' can be set in the VM config. Additionally the 'migrate'
bwlimit from datacenter.cfg (storage-specific limits play no role for
memory+state migration) or the parameter provided to the API call can restrict
the speed. Take the lower of the two.
This patch also refactors the setting of migrate_speed and comments for clarity.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
disk is not copied to the target node but still deleted on cleanup
(phase3_cleanup).
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
commit 4530494bf9 introduced an
regression with local disk migrations if the VM is online and thus
needs to live migrated and no target storage was passed as parameter.
We made the hack to write "1" to the targetstorage option in this
case obsolete, but it was still used on deciding if there are any
drives to mirror at all. Here it is enough to check if there are any
'online_local_volumes' because that hash gets only filled if we can
and are told to live mirror local disk on migrations anyway. Also,
we abort early if local disks are found and the 'with-local-disks'
option is not set.
This was reported at:
https://forum.proxmox.com/threads/livemigration-with-localdisk-doesnt-coppy-and-data-from-the-hdds-anymore.50744/
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the check for targetstorage in:
if ($self->{running} && $self->{opts}->{targetstorage} && $local_volumes->{$volid}->{ref} eq 'config') {
was obsolete, as we always set the tragetstorage opts variable to '1'
in a broader "use same sid for remote local" check above.
So removing it leads to the same if truthtable but fixes the
check if we should fallback to the volume's SID if targetstorage is
not set, as else it seemed to be always set, and '1' is naturally not
a correct stroage ID.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
targetsid was not used, for disk unused (offline copy)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Some storage like rbd or lvm can't keep thin-provising after a qemu-mirror.
Call qga guest-fstrim if qga is available and fstrim_cloned_disks is enabled
after move_disk and migrate.
Co-Authored-By: Alexandre Derumier <aderumier@odiso.com>
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
qemu 2.11 need a power of 2 cache size.
"
Parameter 'xbzrle_cache_size' expects is invalid,
it should be bigger than target page size and a power of two
"
roundup to near power of 2 value
With shared=1, (live) migration ignores the disk and assumes it is
present on all target nodes. This works similar to shared=1 on LXC
mountpoints.
Signed-off-by: Chris Hofstaedtler <chris.hofstaedtler@deduktiva.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This should not be needed since we call 'block-job-complete' before
in qemu_drive_mirror_monitor(), and after benchmarking it does not
appear to be needed nor provide a measurable improvement when shutting
down the source.
and only transfer state and switch direction if there
actually are any replicated volumes.
once we add support for live-migration with replicated
volumes, adding a set-replication-state command to the
tunnel and using that probably makes sense.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
because we want commands to return meaningful errors, and
print them on the client/source side.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
qm mtunnel was deemed as deprecated but still in use here.
Switch over to pvecm's mtunnel to allow removing the qm variant in
PVE 5.1
Also use an absolute path so we do not depended on the targets
environment variables
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This reverts commit 63d02c7074.
The commit changes the configuration before the VM is actually
migrated, so it is possible to have a wrong configuration when
migration fails for some reason. Also, I am quite unsure if
this automatic target change is really wanted. The patch also
contains wrong refereces to $self->{opts}->{node}.
if we define a different target storeid for remote node,
and that storage is not available on source node
Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
since Qemu 2.9, block device write access is limited to one
writer unless shared_rw is set to true. there is an
exception for live-migrating local disks via NBD as long as
the VM is suspended.
stop the NBD server before resuming the VM accordingly to
unbreak local disk live-migration.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
As Fabian as required,
add an extra flag "with-local-disks" to enable live storage migration with localdisk.
default target storage is same sid than source, this can be overrided with
"targetstorage" option.
I will try improve this later, with optionnal mapping, disk by disk.
Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
This allow to migrate disks on local storage to a remote node storage.
When the target node start, a new volumes are created and exposed through qemu embedded nbd server.
qemu drive-mirror is launch on source vm for each disk with nbd server as target.
when drive-mirror reach 100% of 1 disk, we don't complete the block jobs and begin mirror of next disk.
(mirroring are parralel, but we try to mirroring them 1 by 1 to avoid storage && network overload)
Then we live migrate the vm to destination node. (drive-mirror still occur at the same time).
We the vm is livemigrate (source vm paused, target vm pause), we complete the block jobs mirror.
When is done we stop the source vm and resume the target vm
Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
we can migrate local snapshots when on zfs or dir storage with qcow2,
but the check was incorrect
we checked for if (zfs && !qcow2) instead of if (zfs || qcow2)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Without this patch we use the network were the cluster traffic runs
for sending migration traffic. This is not ideal as it may hinder
cluster traffic. Further some users have a powerful network which
would be perfect for migrations, with this patch they can run the
migration traffic over such a network without having the corosync
traffic on the same network.
The network is configurable through /etc/pve/datacenter.cfg which
got a new property, namely migration. migration has two
subproperties: type (replaces the old migration_unsecure property)
and network.
For the case of a network failure or that a VM has to be moved over
another network for arbitrary other reasons I added the
migration_type and migration_network parameters to qm migrate (and
respectively vm_start as this gets used on migration).
They allow overwriting the datacenter.cfg settings.
Fixes bug #1177
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
foreach_volid recurses over snapshots as well, resulting in
lots of repeated checks (especially for VMs with lots of
snapshots and disks).
a potential vmstate volume must be checked explicitly,
because foreach_drive does not care about those.
Restore previous behaviour and do not request a forward tunnel on
insecure migrations.
For the migrations of all kind this has no direct impact, they all
worked, but an port to much requested from an limited pool is still
not ideal. Also an open tunnel, if not needed.
This is a light regression introduced from commit 1c9d54b.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Output all errors - if any - and add some log outputs on what we qmp
commands we do with which parameters, may be helpful when debugging
or analyzing a users problem.
Also check if the queried status is defined, as on a error this may
not be.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
On error let phase2_cleanup close the tunnel as it stops the for
incoming migration waiting VM on the destination first, to be safe.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We cannot guarantee when the SSH forward Tunnel really becomes
ready. The check with the mtunnel API call did not help for this
prolem as it only checked that the SSH connection itself works and
that the destination node has quorum but the forwarded tunnel itself
was not checked.
The Forward tunnel is a different channel in the SSH connection,
independent of the SSH `qm mtunnel` channel, so only if that works
it does not guarantees that our migration tunnel is up and ready.
When the node(s) where under load, or when we did parallel
migrations (migrateall), the migrate command was often started
before a tunnel was open and ready to receive data. This led to
a direct abortion of the migration and is the main cause in why
parallel migrations often leave two thirds or more VMs on the
source node.
The issue was tracked down to SSH after debugging the QEMU
process and enabling debug logging showed that the tunnel became
often to late available and ready, or not at all.
Fixing the TCP forward tunnel is quirky and not straight ahead, the
only way SSH gives as a possibility is to use -N (no command)
-f (background) and -o "ExitOnForwardFailure=yes", then it would
wait in the foreground until the tunnel is ready and only then
background itself. This is not quite the nicest way for our special
use case and our code base.
Waiting for the local port to become open and ready (through
/proc/net/tcp[6]] as a proof of concept is not enough, even if the
port is in the listening state and should theoretically accept
connections this still failed often as the tunnel was not yet fully
ready.
Further another problem would still be open if we tried to patch the
SSH Forward method we currently use - which we solve for free with
the approach of this patch - namely the problem that the method
to get an available port (next_migration_port) has a serious race
condition which could lead to multiple use of the same port on a
parallel migration (I observed this on my many test, seldom but if
it happens its really bad).
So lets now use UNIX sockets, which ssh supports since version 5.7.
The end points are UNIX socket bound to the VMID - thus no port so
no race and also no limitation of available ports (we reserved 50 for
migration).
The endpoints get created in /run/qemu-server/VMID.migrate and as
KVM/QEMU in current versions is able to use UNIX socket just as well
as TCP we have not to change much on the interaction with QEMU.
QEMU is started with the migrate_incoming url at the local
destination endpoint and creates the socket file, we then create
a listening socket on the source side and connect over SSH to the
destination.
Now the migration can be started by issuing the migrate qmp command
with an updated uri.
This breaks live migration from new to old, but *not* from old to
new, so there is a upgrade path.
If a live migration from new to old must be made (for whatever
reason), use the unsecure_migration setting (man datacenter.conf)
to allow this, although that should only be done in trusted network.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
use waitpid with WNO_HANG to check if the ssh tunnel child process
is still running and collect at the same time if it exited.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead, just print a warning if the connection check fails.
as long as the storage is online on the target node, the VM
will start fine after migration.
Drop load_config, write_config, lock_config[_xx],
check_lock, check_protection, is_template and config_file
in favour of implementions in PVE::AbstractConfig.
Implement guest_type, __config_max_unused_disks,
config_file_lock and cfs_config_path from
PVE::AbstractConfig in PVE::QemuConfig.
Since write_config was always called with skiplock=1 except
once, it makes sense to drop this parameter like in
PVE::LXC::write_config . If needed in the future, the
caller can use check_lock before write_config anyway.
