Commit Graph

1102 Commits

Author SHA1 Message Date
Fabian Ebner
cc1a3820db sync_disks: use allow_rename to avoid collisions on the target storage
This makes it possible to migrate a VM with volumes store1:vm-123-disk-0
store2:vm-123-disk-0 to some targetstorage. Also prevents migration failure
when there is an orphaned disk with the same volid on the target.

To avoid confusion, the name should not change for 'vmstate'-volumes.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-08 22:11:54 +02:00
Fabian Ebner
912792e245 Switch to using foreach_volume instead of foreach_drive
It was necessary to move foreach_volid back to QemuServer.pm

In VZDump/QemuServer.pm and QemuMigrate.pm the dependency on
QemuConfig.pm was already there, just the explicit "use" was missing.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-08 22:11:54 +02:00
Thomas Lamprecht
2f6f002c04 config_to_command: indentation, whitspace and newline fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-04-08 17:44:37 +02:00
Stefan Reiter
b516c84871 Ignore version checks when using QEMU -rc releases
Upstream marks these as having a micro-version of >=90, unfortunately the
machine versions are bumped earlier so testing them is made unnecessarily
difficult, since the version checking code would abort on migrations etc...

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
[ Thomas: do so refactor ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-04-08 17:44:37 +02:00
Fabian Grünbichler
e1cfa02e62 config: harmonize bridge pattern
so that pve-container and qemu-server use the same one, in preparation
for moving it to JSONSchema and having a bridgepair format.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-04-08 14:11:33 +02:00
Stefan Reiter
ea1c21108f Include "-cpu" parameter with snapshots/suspend
Just like with live-migration, custom CPU models might change after a
snapshot has been taken (or a VM suspended), which would lead to a
different QEMU invocation on rollback/resume.

Save the "-cpu" argument as a new "runningcpu" option into the VM conf
akin to "runningmachine" and use as override during rollback/resume.

No functional change with non-custom CPU types intended.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-04-07 17:27:58 +02:00
Stefan Reiter
58c64ad5d9 Include "-cpu" parameter with live-migration
This is required to support custom CPU models, since the
"cpu-models.conf" file is not versioned, and can be changed while a VM
using a custom model is running. Changing the file in such a state can
lead to a different "-cpu" argument on the receiving side.

This patch fixes this by passing the entire "-cpu" option (extracted
from /proc/.../cmdline) as a "qm start" parameter. Note that this is
only done if the VM to migrate is using a custom model (which we can
check just fine, since the <vmid>.conf *is* versioned with pending
changes), thus not breaking any live-migration directionality.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-04-07 17:27:58 +02:00
Thomas Lamprecht
b6d9b54b5b map_storage: check for definedness not perl-truthiness
as else it would reject possible a valid set value

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-04-07 17:25:38 +02:00
Fabian Grünbichler
84da82174e vm_start: collect and return migration info
in addition to printing it. preparation for remote cluster migration,
where we want to return this in a structured fashion over the migration
tunnel instead of parsing stdout via SSH.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-04-07 17:07:36 +02:00
Fabian Grünbichler
ba5396b580 vm_start: split NBD disk allocation further
into one sub that retrieves the local disks, and the actual NBD
allocation. that way, remote incoming migration can just call the NBD
allocation with a custom list of volume names/storages/..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-04-07 17:07:32 +02:00
Fabian Grünbichler
bf8fc5a307 migrate: allow arbitrary source->target storage maps
the syntax is backwards compatible, providing a single storage ID or '1'
works like before. the new helper ensures consistent behaviour at all
call sites.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-04-02 17:47:14 +02:00
Fabian Grünbichler
13cfe3b7a7 vm_start: split out NBD disk allocation
as preparation of targetstorage mapping and remote migration. this also
removes re-using of the $local_volumes hash in the original code.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-04-01 18:43:00 +02:00
Fabian Grünbichler
3898a56343 vm_start: split out lock + initial checks
to start breaking up vm_start before extending parts for new migration
features like storage and network mapping.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-04-01 18:41:58 +02:00
Fabian Grünbichler
0c498cca36 vm_start: condense signature
as preparation for refactoring it further. remote migration will add
another 1-2 parameters, and it is already unwieldly enough as it is.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-04-01 17:42:15 +02:00
Fabian Grünbichler
0c4fef3f26 rescan: fix call to foreach_volume
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-04-01 13:02:59 +02:00
Fabian Ebner
ca04977c63 update_disk_config: use config volume iterators instead of loops
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-30 09:59:48 +02:00
Dominik Csapak
818ce80ec1 fix efidisks on storages with minimum sizes bigger than OVMF_VARS.fd
on storages where the minimum size of images is bigger than the real
OVMF_VARS.fd file, they get padded to their minimum size

when using such an image, qemu maps it fully to the vm, but the efi
does not find the vars region and creates a file on the first efi
partition it finds

this breaks some settings in the ovmf, such as resolution

to fix this, we have to specify the size for the pflash, so that
qemu only maps the first n bytes in the vm (this only works for
raw files, not for qcow2)

we also have to use the correct size when converting between storages
in 'clone_disk' (used for move disk and cloning vms) and when
live migrating to different storages

when we now expect that the source image is always correctly used/created
(e.g. raw with size=x in pflash argument) then we always create the
target correctly

when encountering users which have a non-valid image (e.g. a efidisk
moved from zfs to qcow2 before this patch), we have to tell them to
recreate the efidisk and the settings on it

we have to version_guard it to 4.1+pve2 (since we haven't bumped yet
since the change to pve2)

also add 2 tests, one for the old version and one for the new

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Stefan Reiter <s.reiter@proxmox.com>
Reviewed-by: Stefan Reiter <s.reiter@proxmox.com>
[ Thomas: rebased to master ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-03-30 09:41:55 +02:00
Fabian Grünbichler
88126be3f7 migrate: fix replication false-positives
by only checking for replicatable volumes when a replication job is
defined, and passing only actually replicated volumes to the target node
via STDIN, and back via STDOUT.

otherwise this can pick up theoretically replicatable, but not actually
replicated volumes and treat them wrong.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-03-27 07:54:44 +01:00
Stefan Reiter
5d008ad383 Verify VM-specific CPU configs seperately
$cpu_fmt is being reused for custom CPUs as well as VM-specific CPU
settings. The "pve-vm-cpu-conf" format is introduced to verify a config
specifically for use as VM-specific settings.

"pve-cpu-conf" is registered for use in custom CPU API calls (where no
additional checks are required).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-03-25 18:42:24 +01:00
Fabian Ebner
47250f03ef Fix calls to get_replicateable_volumes
There is a need to set $noerr, because otherwise migration for a
VM with a non-replicatable volume fails with:
missing replicate feature on volume 'myfs:107/vm-107-disk-2.raw'

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-25 14:53:17 +01:00
Fabian Grünbichler
9b6efe436d migrate: add live-migration of replicated disks
with incremental drive-mirror and dirty-bitmap tracking.

1.) get replicated disks that are currently referenced by running VM
2.) add a block-dirty-bitmap to each of them
3.) replicate ALL replicated disks
4.) pass bitmaps from 2) to drive-mirror for disks from 1)
5.) skip replicated disks when cleaning up volumes on either source or
target

added error handling is just removing the bitmaps if an error occurs at
any point after 2, except when the handover to the target node has
already happened, since the bitmaps are cleaned up together with the
source VM in that case.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Stefan Reiter <s.reiter@proxmox.com>
2020-03-24 12:22:32 +01:00
Fabian Grünbichler
bc6c823164 drive-mirror: add support for incremental sync
by re-using a dirty bitmap that represents changes since the divergence
of source and target volume. requires a qemu that supports incremental
drive-mirroring, and will die otherwise.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Tested-by: Stefan Reiter <s.reiter@proxmox.com>
2020-03-24 11:54:20 +01:00
Fabian Ebner
43c4c7b693 Add unused description to drivedesc_hash
Moved code so that initialization of drivedesc_hash stays a single block.
Avoid auto-vivication in parse_drive.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-23 09:58:30 +01:00
Stefan Reiter
47f35977cb version_guard: early out when major/minor version is high enough
E.g.: If a feature requires 4.1+pveN and we're using machine version 4.2
we don't need to increase the pve version to N (4.2+pve0 is enough).

We check this by doing a min_version call against a non-existant higher
pve-version for the major/minor tuple we want to test for, which can
only work if the major/minor alone is high enough.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-03-23 09:58:17 +01:00
Fabian Grünbichler
756cd7ea56 vm_start: drop redundant if
since both if and else branch contain identical code

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-03-18 10:08:54 +01:00
Fabian Grünbichler
db1f8b39e1 drive_mirror: rename variables and values
and add some more details to comments.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-03-18 08:21:29 +01:00
Mira Limbeck
eb8cddb55e add NBD server unix socket support in vm_start
As the NBD server spawned by qemu can only listen on a single socket,
we're dependent on a version being passed to vm_start that indicates
which protocol can be used, TCP or Unix, by the source node.

The change in socket type (TCP to Unix) comes with a different URI. For
unix sockets it has the form: 'nbd:unix:<path/to/socket>:exportname=<device>'.

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2020-03-18 08:03:44 +01:00
Mira Limbeck
e02fb12620 add qemu_drive_mirror_monitor completion modes
With Qemu 4.2 we encountered a problem with unix sockets and SSH socket
forwarding for drive-mirror. It seems the socket gets reopened again and
again after it closes for some reason. This can be worked around by
specifying 'block-job-cancel' instead of 'block-job-complete' when we're
not interested in swapping the disks again from NBD to their original
protocol. This is always the case when we use drive-mirror for live
migrating a VM.

qemu_drive_mirror is used for migration and for clone_disk. All in all
we have 3 cases to handle. Either the 'skip' case which skips the
completion of the job. The 'wait' case which was the default before and
still is when $completion is undefined. And the new 'wait_noswap' case
which is used for the live migration.
If 'wait_noswap' is specified, we issue a 'block-job-cancel' once the block
job is in 'ready' state. This completes the block job without swapping the
disks.

clone_disk always uses 'block-job-cancel' via the qemu_blockjobs_cancel
sub.

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2020-03-18 08:03:44 +01:00
Dietmar Maurer
e400f5e523 PVE/QemuServer.pm - proxmox backup server restore: use new pbs-restore binary
This is much faster than using a pipe with qemu-img.
2020-03-11 11:47:24 +01:00
Dietmar Maurer
9f3d73bc35 cleanup backup code: share code between restore_vma_archive and restore_proxmox_backup_archive 2020-03-11 11:47:24 +01:00
Dietmar Maurer
d1e92cf622 implement restore from proxmox backup storage 2020-03-11 11:47:24 +01:00
Alexandre Derumier
28e129cccb sdn: use sdn tap_create|plug
Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-03-10 19:08:31 +01:00
Thomas Lamprecht
81d6e4e1b8 followup: do not import unused methods
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-03-10 14:26:29 +01:00
Oguz Bektas
0f1af9e777 QemuServer: use helper functions from GuestHelpers
removes safe_string_ne and safe_num_ne code which is now shared in
GuestHelpers. also change all the calls to use the shared definitions.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-03-10 09:39:15 +01:00
Fabian Ebner
776c5f5067 Rename disksize to bootdisk_size and print_drive_full to print_drive_commandline_full
to avoid confusion with print_drive

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-07 18:23:57 +01:00
Fabian Ebner
e0fd2b2f84 Create Drive.pm and move drive-related code there
The initialization for the drive keys in $confdesc is changed
to be a single for-loop iterating over the keys of $drivedesc_hash and
the initialization of the unusedN keys is move to directly below it.

To avoid the need to change all the call sites, functions with more than
a few callers are exported from the submodule and imported into QemuServer.pm.

For callers of the now imported functions within QemuServer.pm, the prefix
PVE::QemuServer is dropped, because it is unnecessary and now even confusing.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-07 18:23:57 +01:00
Fabian Ebner
ad396d54e4 Use parse_drive for EFI disk
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-07 18:23:57 +01:00
Fabian Ebner
28438925dc Replace drivename_hash with drivedesc_hash
which contains the full descriptions of the drives, and
make parse_drive not depend on $confdesc anymore.
In preparation to moving drive-related code to its own module.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-03-07 18:23:57 +01:00
Stefan Reiter
2cf61f33d9 fix #2264: add virtio-rng device
Allow a user to add a virtio-rng-pci (an emulated hardware random
number generator) to a VM with the rng0 setting. The setting is
version_guard()-ed.

Limit the selection of entropy source to one of three:
/dev/urandom (preferred): Non-blocking kernel entropy source
/dev/random: Blocking kernel source
/dev/hwrng: Hardware RNG on the host for passthrough

QEMU itself defaults to /dev/urandom (or the equivalent getrandom()
call) if no source file is given, but I don't fully trust that
behaviour to stay constant, considering the documentation [0] already
disagrees with the code [1], so let's always specify the file ourselves.

/dev/urandom is preferred, since it prevents host entropy starvation.
The quality of randomness is still good enough to emulate a hwrng, since
a) it's still seeded from the kernel's true entropy pool periodically
and b) it's mixed with true entropy in the guest as well.

Additionally, all sources about entropy predicition attacks I could find
mention that to predict /dev/urandom results, /dev/random has to be
accessed or manipulated in one way or the other - this is not possible
from a VM however, as the entropy we're talking about comes from the
*hosts* blocking pool.

More about the entropy and security implications of the non-blocking
interface in [2] and [3].

Note further that only one /dev/hwrng exists at any given time, if
multiple RNGs are available, only the one selected in
'/sys/devices/virtual/misc/hw_random/rng_current' will feed the file.
Selecting this is left as an exercise to the user, if at all required.

We limit the available entropy to 1 KiB/s by default, but allow the user
to override this. Interesting to note is that the limiter does not work
linearly, i.e. max_bytes=1024/period=1000 means that up to 1 KiB of data
becomes available on a 1000 millisecond timer, not that 1 KiB is
streamed to the guest over the course of one second - hence the
configurable period.

The default used here is the same as given in the QEMU documentation [0]
and has been verified to affect entropy availability in a guest by
measuring /dev/random throughput. 1 KiB/s is enough to avoid any
early-boot entropy shortages, and already has a significant impact on
/dev/random availability in the guest.

[0] https://wiki.qemu.org/Features/VirtIORNG
[1] https://git.qemu.org/?p=qemu.git;a=blob;f=crypto/random-platform.c;h=f92f96987d7d262047c7604b169a7fdf11236107;hb=HEAD
[2] https://lwn.net/Articles/261804/
[3] https://lwn.net/Articles/808575/

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-03-06 18:09:04 +01:00
Fabian Ebner
d7117b5adb print_drive: Use $skip to avoid the need to copy the hash
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-02-25 12:29:38 +01:00
Fabian Ebner
375db731f7 Align size to 1 KiB bytes before doing 'qmp block_resize'
1. Avoids the error
"VM 111 qmp command 'block_resize' failed - The new size must be a multiple of 512"
for qcow2 disks.
2. Because volume_import expects disk sizes to be a multiple of 1 KiB.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-02-19 15:31:41 +01:00
Stefan Reiter
a04dd5c455 Simplify QEMU version check and require 3.0+
Some of the recent QMP changes require at least 2.8.0, but since the
oldest version we officially package for 6.x is 4.0.0 anyway, checking
for at least 3.0 should not break anyone's setup.

Note that this does not affect machine version checks, only the
installed QEMU binary version.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-02-12 11:19:29 +01:00
Stefan Reiter
b8fb1c03c3 version_guard scsi drive count
Live-migrating a VM with more than 14 SCSI disks to a node that doesn't
support it yet is broken. Use a bumped pve-version to represent that and
give the user a nice error message instead.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-02-12 10:32:57 +01:00
Stefan Reiter
ac0077cc33 Use 'QEMU version' -> '+pve-version' mapping for machine types
The previously introduced approach can fail for pinned versions when a
new QEMU release is introduced. The saner approach is to use a mapping
that gives one pve-version for each QEMU release.

Fortunately, the old system has not been bumped yet, so we can still
change it without too much effort.

QEMU versions without a mapping are assumed to be pve0, 4.1 is mapped to
pve1 since thats what we had as our default previously.

Pinned machine versions (i.e. pc-i440fx-4.1) are always assumed to be
pve0, for specific pve-versions they'd have to be pinned as well (i.e.
pc-i440fx-4.1+pve1).

The new logic also makes the pve-version dynamic, and starts VMs with
the lowest possible 'feature-level', i.e. if a feature is only available
with 4.1+pve2, but the VM isn't using it, we still start it with
4.1+pve0.

We die if we don't support a version that is requested from us. This
allows us to use the pve-version as live-migration blocks (i.e. bumping
the version and then live-migrating a VM which uses the new feature (so
is running with the bumped version) to an outdated node will present the
user with a helpful error message and fail instead of silently modifying
the config and only failing *after* the migration).

$version_guard is introduced in config_to_command to use for features
that need to check pve-version, it automatically handles selecting the
newest necessary pve-version for the VM.

Tests have to be adjusted, since all of them now resolve to pve0 instead
of pve1. EXPECT_ERROR matching is changed to use 'eq' instead of regex
to allow special characters in error messages.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-02-12 10:32:57 +01:00
Stefan Reiter
65af8c312e qmp: use query-cpus-fast for hotplug
query-cpus has been deprecated since 2.12.0 [0] in favor of
query-cpus-fast, which no longer incurs a guest performance penalty on
the guest. The returned information is the same as far as our use case
is concerned.

[0] https://qemu.weilnetz.de/doc/qemu-doc.html#Deprecated-features

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-02-06 13:50:33 +01:00
Stefan Reiter
922869f223 qmp: use 'id' parameter instead of 'device'
'device' is deprecated since 2.8 in favor of 'id' [0], but since we
always consistently set the id on our drives anyway we can substitute it
easily.

[0] see files qapi/block.json and qapi/block-core.json in QEMU source
code, the online documentation doesn't mention it AFAICT

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-02-06 13:50:33 +01:00
Stefan Reiter
ce9fce7985 qmp: use 'blockdev-change-medium' instead of generic 'change'
...and cleanup surrounding code a bit.

'change' is deprecated, and according to the qapi definition in QEMU it
is 'strongly recommended' to avoid using it.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-02-06 13:50:33 +01:00
Stefan Reiter
9e7bce2c88 Remove unused $force parameter
The fixme got it right :)

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-02-06 13:50:33 +01:00
Fabian Ebner
8b02e56870 rename 'volid' to 'drivestr' where it's not only a volume ID
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-02-05 08:41:05 +01:00
Dominik Csapak
2513b862e6 fix #2566: increase scsi limit to 31
to achieve this we have to add 3 new scsihw addresses since lsi
controllers can only hold 7 scsi drives

we go up to 31, since this is the limit for virtio-scsi-single devices
we have reserved (we can increase this in the future)

to make it more future proof, we add a new pci bridge under pci
bridge 1, so we have to adapt the bridge adding code (we did not
need this for q35 previously)

impact on live migration:
since on older versions of qemu-server we do not have those config
settings, there is no problem from old -> new

new->old is not supported anyway and this breaks so that
the vm crashes and loses the configs for scsi15-30
(same behaviour as e.g. with audio0 and migration from new->old)

tested with 31 scsi disk on
i440fx + virtio-scsi
i440fx + lsi
q35 + virtio-scsi
q35 + lsi
with ovmf + seabios

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-01-31 20:26:26 +01:00
Dominik Csapak
844d8fa628 move the vmgenid device after readconfig on q35
and adapt the tests

this does not impact live migration, since the order here does not
change the device layout

we want this to consistently have the readconfig first

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-01-31 20:26:26 +01:00
Oguz Bektas
74f15a8993 hotplug_pending: make 'ssd' option non-hotpluggable
from hotplug_pending we go into 'vmconfig_update_disk', where we check the
hotpluggability of options.

add 'ssd' there as a non-hotpluggable option (since we'd have to unplug/plug to
change the drive type)

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-24 16:11:53 +01:00
Stefan Reiter
d786a27435 Add CPUConfig file and migrate some helpers
The package will be used for custom CPU models as a SectionConfig, hence
the name. For now we simply move some CPU related helper functions and
declarations over from QemuServer to reduce clutter there.

Exports are to avoid changing all call sites, functions have useful
names on their own.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-01-22 15:47:32 +01:00
Fabian Ebner
301c675281 Use 'volname' instead of 'volid' for 'qemu_img_format'
As 'qemu_img_format' just matches a regex, this doesn't make much of
a difference, but AFAICT all other calls of 'qemu_img_format' use 'volname'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-01-21 07:45:31 +01:00
Fabian Ebner
c3c5d2b6b3 Fix 2070: vm_start: for a migrating VM, use current format of disk if possible
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-01-21 07:45:31 +01:00
Oguz Bektas
3d48b95aa8 vmconfig_apply_pending: remove redundant write/load config calls
since we handle errors gracefully now, we don't need to write & save
config every time we change a setting.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-16 10:29:31 +01:00
Tim Marx
ef3f42930f add timeout parameter to vm_start api endpoint
Signed-off-by: Tim Marx <t.marx@proxmox.com>
2020-01-15 17:36:16 +01:00
Tim Marx
2f18c84dc7 add new helper to calculate timeout based on vm config
Signed-off-by: Tim Marx <t.marx@proxmox.com>
2020-01-15 17:36:16 +01:00
Stefan Reiter
52cffab6b5 Fixup CPU flag query to not use get_basic_machine_info
get_basic_machine_info was removed by commit 045749f2fc.

Use get_host_arch to get the default machine type instead, and
optionally allow to specify architecture as parameter.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-01-15 15:57:00 +01:00
Stefan Reiter
05a4c550f3 Add QEMU CPU flag querying helpers
* query_understood_cpu_flags returns all flags that QEMU/KVM knows about
* query_supported_cpu_flags returns all flags that QEMU/KVM can use on
  this particular host.

To get supported flags, a temporary VM is started with QEMU, so we can
issue the "query-cpu-model-expansion" QMP command. This is how libvirt
queries supported flags for its "host-passthrough" CPU type.
query_supported_cpu_flags is thus rather slow and shouldn't be called
unnecessarily.

Note that KVM and TCG accelerators provide different expansions for the
"host" CPU type, so we need to query both.

Currently only supports x86_64, because QEMU-aarch64 doesn't provide the
necessary querying functions.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-01-14 11:56:21 +01:00
Oguz Bektas
eb5e482ded vmconfig_apply_pending: add error handling
wrap around code which can possibly fail in evals to handle them
gracefully, and log errors.

note: this results in a change of behavior in the API. since errors
are handled gracefully instead of "die"ing, when there is a pending
change which cannot be applied for some reason, it will get logged in
the tasklog but the vm will continue booting regardless. the
non-applied change will stay in the pending section of the
configuration.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-01-14 11:50:34 +01:00
Oguz Bektas
4df15a0377 hotplug_pending: remove redundant write/load config calls
instead of writing the config after every change, we can do it once for
all the changes in the end to avoid redundant i/o.

we also don't need to load_config after writing fastplug changes.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2019-12-20 10:51:52 +01:00
Thomas Lamprecht
ae200950d4 grammar fix: s/does not exists/does not exist/g
bump versioned build-dependency, as qemu-server has tests checking
for errors, and we fixed an grammar error in pve-storage, so we need
the newer version to ensure our test go through

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-13 12:20:56 +01:00
Thomas Lamprecht
128193e26a followup: simplify log callback
run_command only passes defined and chomped strings to the callback,
so no need to do that twice.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-12 13:39:04 +01:00
Stefan Reiter
8bf30c2a72 fix #2493: show QEMU errors in migration log
QEMU usually only prints warnings and errors and stays silent otherwise,
so it makes sense to just log all of it's output.

Prefix it with '[<target_hostname>]' to indicate that the output is
coming from the remote node, so users know where to search for the
error.

Side effect is that the 'VM start' task created by the migration will
now show the "QEMU:" prefix, but it's still very readable IMHO.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-12-12 13:36:19 +01:00
Stefan Reiter
6e0216d862 hide long commandline on vm_start/migrate failure
By default run_command prints the entire commandline executed when an
error occurs, but QEMU and our migrate command are not only
uninteresting to the user[*] but also annoyingly long. Hide them and only
print the exit code.

[*] Especially our migrate command, since it can't be manually executed
anyway. QEMU's commandline *might* contain something interesting, but is
so long that it's tricky to parse anyway, any a user can always call 'qm
showcmd --pretty'.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-12-12 13:35:40 +01:00
Stefan Reiter
68b108ee3a update disk size before local disk migration
Split out 'update_disksize' from the renamed 'update_disk_config' to
allow code reuse in QemuMigrate.

Remove dots after messages to keep style consistent for migration log.

After updating in sync_disks (phase1) of migration, write out updated
config. This means that even if migration fails or is aborted in later
stages, we keep the fixed config - this is not an issue, as it would
have been fixed on the next attempt anyway, and it can't hurt to have
the correct size instead of a wrong one either way.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-12-11 10:42:56 +01:00
Dominik Csapak
75c24bba0a suspend to disk: check more permissions
only VM.PowerMgmt is not enough, since we allocate space on a storage,
so we need VM.Config.Disk on the vm and Datastore.AllocateSpace on the storage

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-12-11 09:02:55 +01:00
Dominik Csapak
66cebc4671 refactor finding of vmstate storage
we need that on another place, so refactor in its own sub

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-12-11 09:02:55 +01:00
Thomas Lamprecht
38277afcd4 qemu-server: make nodename mock-able for tests
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-10 11:08:33 +01:00
Thomas Lamprecht
cbb39a7e79 followup: improve error message for outdated QEMU version
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-10 11:08:33 +01:00
Stefan Reiter
a77a53ae54 check if QEMU version is recent enough for machine type
...and show the user a more meaningful error otherwise.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-12-10 09:13:33 +01:00
Stefan Reiter
71c58bb7ed remove $vmid param from print_drive
It isn't used in the sub, but suggest it is needed. No users outside
qemu-server found.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-12-09 11:44:13 +01:00
Thomas Lamprecht
c7728fdb85 followup: minor cleanup/error message spelling
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-09 11:30:14 +01:00
Dominik Csapak
844b55fb89 fix #2510: hostpci: always check if device exists
if the user set a device as hostpci with the 'shorthand' syntax:

hostpciX: 00:12

we ignored it on starting and showcmd and continued.
Since the user explicitly wanted to passthrough a device, we now check
if there is actually a device with that id

for explicitly configured devices (00:12.1), we did not check if it exists,
but the kvm call failed with a non-obvious error message

now we always call 'lspci' from SysFSTools to check if it actually exists,
and fail if not. With this, we can drop the workaround for adding
'0000' if no domain was given, since lspci does it already for us

this fixes #2510, an issue with using mediated devices where the users did not have
the domain in the config, since we forgot to add the default domain there

the only issue with this patch is that it changes the behaviour of
'showcmd' slightly, as in now, we die if the device was explicitly
given, but did not exists (we showed the commandline, now we fail)

this also slightly changes the commandline for qemu (adding always
the domain), which is not a problem since we cannot live migrate
or snapshot such vms, but we have to adapt the tests

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-12-09 11:30:14 +01:00
Thomas Lamprecht
bdd1feef5b fix #2469: fix qemu-img convert src_format detection
This reverts commit c5151cb8bb which is
a revert of the wrongly done revert of
commit e2414e73ce.
2019-12-09 10:31:33 +01:00
Thomas Lamprecht
340dbcf730 create_efidisk: poll the real size after volume creation
some storage backends have bigger granularity than the default 128k
size from the EFIVARS template file, so we actually need to poll the
real created disk size, as it will be used to create the target
volume for local storage migration on running VMs, if it's to small
the target will be to small and migration will fail.

Just a fix for newly created EFIDISKS, for others we need to rescan
the size after we've got the migrate lock and write the updated info
out, so that the target node has the correct one (protected from
migrate lock).

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-04 18:42:27 +01:00
Dominik Csapak
ad1d49f45c delete suspended lock when we have no vmstate
if a user removed the vmstate from the config for whatever reason,
a vmstart did not remove the 'suspended' lock
so always delete it and delete the vmstate only if it really was there

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-11-30 17:28:23 +01:00
Dominik Csapak
b85666cf7f logging vmstate into Resume/Start Task
so that the user knows what got loaded as vmstate

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-11-30 17:28:23 +01:00
Oguz Bektas
7a4bdb3662 skip efidisk0 in hotplug
while it's a disk from our storage POV, in QEMU it's a pflash, and
those cannot be hot-plugged

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-26 17:11:49 +01:00
Thomas Lamprecht
c5151cb8bb Revert "fix #2469: fix qemu-img convert src_format detection"
This reverts commit e2414e73ce.
2019-11-26 13:06:57 +01:00
Thomas Lamprecht
9471e48bf9 implement PVE Version addition for QEMU machine
With our QEMU 4.1.1 package we can pass a additional internal version
to QEMU's machine, it will be split out there and ignored, but
returned on a QMP 'query-machines' call.

This allows us to use it for increasing the granularity with which we
can roll-out HW layout changes/additions for VMs. Until now we
required a machine version bump, happening normally every major
release of QEMU, with seldom, for us irrelevant, exceptions.
This often delays rolling out a feature, which would break
live-migration, by several months. That can now be avoided, the new
"pve-version" component of the machine can be bumped at will, and
thus we are much more flexible.

That versions orders after the ($major, $minor) version components
from an stable release - it can thus also be reset on the next
release.

The implementation extends the qemu-machine REGEX, remembers
"pve-version" when doing a "query-machines" and integrates support
into the min_version and extract_version helpers.

We start out with a version of 1.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Stefan Reiter <s.reiter@proxmox.com>
2019-11-25 16:43:38 +01:00
Fabian Grünbichler
e2414e73ce fix #2469: fix qemu-img convert src_format detection
if we don't know which format the source volume/file has, let qemu-img
decide.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-25 13:45:46 +01:00
Fabian Grünbichler
d184401e9f remove wrong comment
the change to encode pending changes differently is not yet included in
qemu-server.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-11-25 10:38:15 +01:00
Oguz Bektas
15c6e277a4 fix #2367: do not allow snapshot with name PENDING
or any other variant of the word 'pending'.

note that we can actually allow this snapshot after PVE 7.0, since
pending section and snapshots will be properly namespaced.
([pve:pending] and [snap:$snapname] or similar).

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2019-11-25 10:01:41 +01:00
Thomas Lamprecht
045749f2fc split get_basic_machine_info into get_vm_arch and get_vm_machine
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-25 08:56:58 +01:00
Thomas Lamprecht
eec4f394f9 really fallback to PCI standard domain "0000" again
fixes commit 70740c1a4d

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-23 09:50:54 +01:00
Dominik Csapak
b8e7068a11 add 'tags' option
this is useful as meta information for e.g., provisioning or config
management systems

adding the info also to the 'status' api call to make it easier to show
it in the gui

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-11-20 20:24:49 +01:00
Dominik Csapak
70740c1a4d fix #2436: pci: do not hardcode pci domain to 0000
relax the regex for hostpci to allow different pci domains than 0000

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-11-20 19:07:46 +01:00
Thomas Lamprecht
6c9f59c122 cleanup: improve locallity of variable declaration/use
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-20 19:06:15 +01:00
Stefan Reiter
2ea5fb7ecf refactor: split qemu_machine_feature_enabled
...into:

* PVE::QemuServer::Helpers::min_version: check a major.minor version
  string with a given major/minor version (this is equivalent to calling
  the old qemu_machine_feature_enabled with only $kvmver)
* PVE::QemuServer::Machine::extract_version: get major.minor version
  string from arbitrary machine type (e.g. pc-q35-4.0, ...)
* PVE::QemuServer::Machine::machine_version: helper to call
  extract_version automatically before min_version

Includes a cfg2cmd test case with pinned machine version.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-20 18:32:35 +01:00
Stefan Reiter
3392d6cacf refactor: extract QEMU machine related helpers to package
...PVE::QemuServer::Machine.

qemu_machine_feature_enabled is exported since it has a *lot* of users
in PVE::QemuServer and a long enough name as it is.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-11-20 16:29:23 +01:00
Stefan Reiter
0a13e08ec2 refactor: create QemuServer::Monitor for high-level QMP access
QMP and monitor helpers are moved from QemuServer.pm.

By using only vm_running_locally instead of check_running, a cyclic
dependency to QemuConfig is avoided. This also means that the $nocheck
parameter serves no more purpose, and has thus been removed along with
vm_mon_cmd_nocheck.

Care has been taken to avoid errors resulting from this, and
occasionally a manual check for a VM's existance inserted on the
callsite.

Methods have been renamed to avoid redundant naming:
* vm_qmp_command -> qmp_cmd
* vm_mon_cmd -> mon_cmd
* vm_human_monitor_command -> hmp_cmd

mon_cmd is exported since it has many users. This patch also changes all
non-package users of vm_qmp_command to use the mon_cmd helper. Includes
mocking for tests.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-11-20 16:29:23 +01:00
Stefan Reiter
babf613a08 refactor: split check_running into _exists_ and _running_
vm_exists_on_node in PVE::QemuConfig checks if a config file for a vmid
exists

vm_running_locally in PVE::QemuServer::Helpers checks if a VM is running
on the local machine by probing its pidfile and checking /proc/.../cmdline

check_running is left in QemuServer for compatibility, but changed to
simply call the two new helper functions.

Both methods are also correctly mocked for testing snapshots.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-11-20 16:29:23 +01:00
Stefan Reiter
24622ebed4 Change check_cmdline to parse_cmdline
parse_cmdline is required for upcoming changes related to custom CPU
types and live migration, and this way we can re-use existing code.

Provides the necessary infrastructure to parse QEMU /proc/.../cmdline.
Changing the single user (check_running) is trivial too.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-11-20 16:29:23 +01:00
Stefan Reiter
d036e418a8 refactor: create QemuServer::Helpers and move file/dir code
Also remove unused $confdir variable in QemuConfig, but leave it and
$lock_dir there, since those paths should only be used with
cfs_config_path anyway.

nodename() is still called in multiple places, but since it's cached by
INotify it doesn't really matter.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-11-20 16:29:23 +01:00
Stefan Reiter
60f03a1168 Fix 'uninitialized value' warning about guest agent type
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-11-20 16:11:44 +01:00
Stefan Reiter
092868c4d1 Add 'runningmachine' support to vm_commandline
to allow 'qm showcmd --snapshot ...' to show the correct machine type.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2019-11-20 16:11:44 +01:00
Thomas Lamprecht
5084a85941 qemu: remove unused methods
found no use with tree-wide search, so remove:
* nic_models
* os_list_description

Both were introduced before the import to SVN happened.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-20 15:37:01 +01:00
Thomas Lamprecht
9e8976ea4e followup: adding qxl max_outputs breaks live-migration/restore
enforce QEMU 4.1.x for that change

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-11-20 15:33:48 +01:00