This can happen after a hard failure, e.g. if the vzdump task was
killed. The next backup (after unlocking the VM) would then fail with
> ERROR: VM 125 qmp command 'backup' failed - previous backup not finished
During the failure path of that attempt, 'backup-cancel' is executed
and the subsequent attempt would then work again. Do it up-front with
a warning instead of relying on this behavior.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
In preparation to use it to conditionally issue a QMP 'backup-cancel'
should a previous backup still be running.
While at it, avoid using the compat-only check_running() helper.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
This moves all error output to stderr while at it and fixes some bad
references to wrong paths in some error messages.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
do not propagate that absolute mess of mixing tabs and spaces to new
programs that ain't perl and thus doesn't need to suffer.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Implement a C program that extracts AMD SEV hardware information such
as reduced-phys-bios and cbitpos from CPUID, looks if SEV, SEV-ES &
SEV-SNP are enabled, and outputs these details as JSON to
/run/qemu-server/host-hw-capabilities.json
This program can also be used to read and save other hardware
information.
Signed-off-by: Markus Frank <m.frank@proxmox.com>
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
When the VM is only started for backup, the VM will be stopped at that
point again. While the detach helpers do not warn about errors
currently, that might change in the future. This is also in
preparation for other cleanup QMP helpers that are more verbose about
failure.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Since pve-common commit:
eff5957 (sysfstools: file_write: properly catch errors)
this check here fails now when the reset does not work. It turns out
that resetting the device is not always necessary, and we previously
ignored most errors when trying to do so.
To restore that functionality, downgrade this `die` to a warning.
If the device really needs a reset to work, it will either fail later
during startup, or not work correctly in the guest, but that behavior
existed before and is AFAIK not really detectable from our side.
Also improve the warning message a bit to not scare users and explain
that we're continuing.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ TL: fine-tune error message a bit and avoid parenthesis ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Adds a syslog entry to log the process id that has been given to the
QEMU VM process at start. This is helpful debugging information if the
pid shows up at other places, like a kernel stack trace, while the VM
has been running, but cannot be retrieved anymore (e.g. the pidfile has
been deleted or only the syslog is available).
The syslog has been put in the `PVE::QemuServer::vm_start_nolock`
subroutine to make sure that the PID is logged not only when the VM has
been started by the API endpoint `vm_start`, but also when the VM is
started by a remote migration.
Suggested-by: Hannes Dürr <h.duerr@proxmox.com>
Suggested-by: Friedrich Weber <f.weber@proxmox.com>
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
Reviewed-by: Daniel Herzig <d.herzig@proxmox.com>
To ensure the new behavior of our sysfs related helper is available
for the changes in commit a28e6fe ("pci: make variable name slightly
easier to read")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Since kernel 6.8, NVIDIAs vGPU driver does not use the generic mdev
interface anymore, since they relied on a feature there which is not
available anymore. IIUC the kernel [0] recommends drivers to implement
their own device specific features since putting all in the generic one
does not make sense.
They now have an 'nvidia' folder in the device sysfs path, which
contains the files `creatable_vgpu_types`/`current_vgpu_type` to
control the virtual functions model, and then the whole virtual function
has to be passed through (although without resetting and changing to the
vfio-pci driver).
This patch implements changes so that from a config perspective, it
still is an mediated device, and we map the functionality iff the device
has no mediated devices but the new NVIDIAs sysfsapi and the model name
is 'nvidia-<..>'
It behaves a bit different than mdevs and normal pci passthrough, as we
have to choose the correct device immediately since it's bound to the
pciid, but we must not bind the device to vfio-pci as the NVIDIA driver
implements this functionality itself.
When cleaning up, we iterate over all reserved devices (since for a
mapping we can't know at this point which was chosen besides looking at
the reservations) and reset the vgpu model to '0', so it frees up the
reservation from NVIDIAs side. (We also do that in a loop, since it's
not always immediately ready after QEMU closes)
A general problem (but that was previously also the case) is that a
showcmd (for a not running guest) reserves the pciids, which might block
an execution of a different real vm. This is now a bit more problematic
as we (temporarily) set the vgpu type then.
0: https://docs.kernel.org/driver-api/vfio-pci-device-specific-driver-acceptance.html
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add an optional parameter to the helper that removes PCI reservations
so that we can partially release IDs again. This will be necessary for
NVIDIAs new sysfs api
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Since the only way this could happen is when we're being called
from 'qm showcmd' and there we don't want to reserve or create anything.
In case the VM was not running, we actually reserve the devices, so we
want to call 'cleanup_pci_devices' after to remove those again. This
minimizes the timespan where those devices are not available for real vm
starts.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
clarify a few units and avoid "since the process start" as it's not
really clear which process is meant and "since the guest was started"
is telling enough too, and as we do a full stop+start cycle on CT
reboot it's true for that too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
i omitted the 'disk' property, since it's non functional currently,
since we don't query the disk usage here (complicated to calculate,
depending on the storage, or requires guest agent support, which is also
non-trivial)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ TL: avoid having netin twice, change to netout once ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This includes docs, and strings printed to stderr or stdout.
These were caught with:
typos --exclude test --exclude changelog
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
As reported in the community forum [0], when a remote migration
request comes in via an API client, the -T flag for Perl is set, so an
insecure dependency in a call like unlink() in forward_unix_socket()
will fail with:
> failed to write forwarding command - Insecure dependency in unlink while running with -T switch
To fix it, untaint the problematic socket addresses coming from the
remote side. Require that all sockets are below '/run/qemu-server/'
and end with '.migrate' with the main socket being matched more
strictly. This allows extensions in the future while still being quite
strict.
[0]: https://forum.proxmox.com/threads/123048/post-691958
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
The version of the running QEMU binary is not related to the machine
version and so it's a bit confusing to have the helper in the
'Machine' module. It cannot live in the 'Helpers' module, because that
would lead to a cyclic inclusion Helpers <-> Monitor. Thus,
'QMPHelpers' is chosen as the new home.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
There is a possibility that the drive-mirror job is not yet done when
the migration wants to inactivate the source's blockdrives:
> bdrv_co_write_req_prepare: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed.
This can be prevented by using the 'write-blocking' copy mode (also
called active mode) for the mirror. However, with active mode, the
guest write speed is limited by the synchronous writes to the mirror
target. For this reason, a way to start out in the faster 'background'
mode and later switch to active mode was introduced in QEMU 8.2.
The switch is done once the mirror job for all drives is ready to be
completed to reduce the time spent where guest IO is limited.
The loop waiting for actively-synced to become true is not an endless
loop: Once the remaining dirty parts have been mirrored by the
background iteration, the actively-synced flag will be set. Because
the 'block-job-change' QMP command already succeeded, new writes will
be done synchronously to the target and thus not lead to new dirty
parts. If the job fails or vanishes (shouldn't actually happen,
because auto-dismiss is false), the loop will be exited and the error
propagated.
Reported rarely, but steadily over the years:
https://forum.proxmox.com/threads/78954/post-353651https://forum.proxmox.com/threads/78954/post-380015https://forum.proxmox.com/threads/100020/post-431660https://forum.proxmox.com/threads/111831/post-482425https://forum.proxmox.com/threads/111831/post-499807https://forum.proxmox.com/threads/137849/
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Currently, when completing a drive mirror job, only errors matching
"cannot be completed" will be handled. Other errors are ignored and
a wrong message that the job was completed successfully will be
printed to the log. An instance of this popped up in the community
forum [0].
The QMP command used for completing the job is either
'block-job-complete' or 'block-job-cancel'. The former causes the VM
to switch to the target drive, the latter doesn't, e.g. migration uses
the latter to not switch the source instance over to the target drive.
The 'block-job-cancel' command doesn't even have the same "cannot be
completed" message, but returns immediately.
The timeout for both 'block-job-cancel' and 'block-job-complete' is
set to 10 minutes in the QMPClient module, which should be enough.
[0]: https://forum.proxmox.com/threads/151518/
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Cloudbase-Init, a cloud-init reimplementation for Windows, supports only
a subset of the configuration options of cloud-init. Some features
depend on support by the Metadata Service (ConfigDrive2 here) and have
further limitations [0].
To support a basic setup the following changes were made:
- password is saved as plaintext for any Windows guests (ostype)
- DNS servers are added to each of the interfaces
- SSH public keys are passed via metadata
Network and metadata generation for Cloudbase-Init is separate from the
default ConfigDrive2 one so as to not interfere with any other OSes that
depend on the current ConfigDrive2 implementation.
DNS search domains were removed because Cloudbase-Init's ENI parser
doesn't handle it at all.
The password set via `cipassword` is used for the Admin user configured
in the cloudbase-init.conf in the guest while the `ciuser` parameter is
ignored. The Admin user has to be set in the cloudbase-init.conf file
instead.
Specifying a different user does not work.
For the password to work the `ostype` needs to be any Windows variant
before `cipassword` is set. Otherwise the password will be encrypted and
the encrypted password used as plaintext password in the guest.
The `citype` needs to be `configdrive2`, which is the default for
Windows guests, for the generated configs to be compatible with
Cloudbase-Init.
[0] https://cloudbase-init.readthedocs.io/en/latest/index.html
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
URI is used in multiple files:
PVE/API2/Qemu.pm
PVE/CLI/qm.pm
PVE/QemuServer.pm
PVE/QemuServer/Cloudinit.pm
Dependencies of qemu-server already have it as dependency, but there's
no explicit dependency in qemu-server yet.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
As reported in the community forum [0], after migration, the VM might
not immediately be able to respond to QMP commands, which means the VM
could fail to resume and stay in paused state on the target.
The reason is that activating the block drives in QEMU can take a bit
of time. For example, it might be necessary to invalidate the caches
(where for raw devices a flush might be needed) and the request
alignment and size of the block device needs to be queried.
In [0], an external Ceph cluster with krbd is used, and the initial
read to the block device after migration, for probing the request
alignment, takes a bit over 10 seconds[1]. Use 60 seconds as the new
timeout to be on the safe side for the future.
All callers are inside workers or via the 'qm' CLI command, so bumping
beyond 30 seconds is fine.
[0]: https://forum.proxmox.com/threads/149610/
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
When detaching and attaching the network device on update, the
link_down setting is not considered and the network device always gets
attached to the guest - even if link_down is set.
Fixes: 3f14f206 ("nic online bridge/vlan change: link disconnect/reconnect")
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Systemd reapplies its known values on reload, so we cannot simply call
into PVE::CGroup. Call systemd's SetUnitProperties method via dbus
instead.
The hotplug and startup code also calculated different values, as one
operated within systemd's value framework (documented in
systemd.resource-control(5)) and one worked with cgroup values
(distinguishing between cgroup v1 and v2 manually).
This is now unified by overriding `change_cpu_quota()` and
`change_cpu_shares()` via `PVE::QemuServer::CGroup` which now takes
systemd-based values and sends those directly via dbus.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
With the last change in the permission check, I accidentally broke the
check for 'spice' host value, since in the if/elsif/else this will fall
through to the else case which was only intended for when neither 'host'
nor 'mapping' was set.
This made 'spice' only settable by root@pam since there we return early.
To fix this, move the spice check into the 'host' branch, but only error
out in case it's not spice.
Fixes: e3971865 (enable cluster mapped USB devices for guests)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Updating the NIC of a VM when the following conditions were met:
* VM is turned off
* NIC is on a bridge that uses automatic dhcp
* Leave bridge unchanged
led to duplicate IPAM entries for the same network device.
This is due to the fact that the add_next_free_cidr always ran on
applying pending network changes.
Now we only add a new ipam entry if either:
* the value of the bridge or mac address changed
* the network device has been newly added
This way no duplicate IPAM entries should get created.
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
tpmstate0 is already included in `get_vm_volumes`, and our only storage
plugin that has unmap_volume implemented is the RBDPlugin, where we call
unmap in `deactivate_volume`. So it's already ummapped by the
`deactivate_volumes` calls above.
For third-party storage plugins, it's natural to expect that
deactivate_volume() would also remove a mapping for the volume just
like RBDPlugin does.
While there is an explicit map_volume() call in start_swtpm(), a
third-party plugin might expect an explicit unmap_volume() call too.
However, the order of calls right now is
1. activate_volume()
2. map_volume()
3. deactivate_volume()
4. unmap_volume()
Which seems like it could cause problems already for third-party
plugins relying on an explicit unmap call.
All that said, it is unlikely that a third-party plugin breaks. If it
really happens, it can be discussed/adapted to the actual needs still.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Acked-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
In Proxmox VE 8, the oldest supported QEMU version is 8.0, so a check
for version 4.0.1 is not required anymore.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
In Proxmox VE 8, the oldest supported QEMU version is 8.0, so a
check for version 4.2 is not required anymore. The check was also
wrong, because it checked the installed version and not the currently
running one.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
While archives with unknown or undetermined subtype could be shown,
this is only for autocompletion, so users can still specify those
manually if required.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Some callers like the move disk API endpoint do not pass an explicit
completion argument. This is not an issue in general, because
qemu_drive_mirror_monitor() defaults to 'complete'. However, there was
a string comparision for the cloudinit case that can trigger a warning
about the value being uninitialized.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
this was a stray search and replace for job -> job_id that should have only
changed variable names..
Fixes: 0ea24bf ("mirror monitor: refactoring/code cleanup")
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
upon failure. Otherwise, the job would disappear too quickly from the
job list and cannot be queried for the actual error anymore.
Relevant part of the error in an actual example:
Before:
> VM 106 qmp command 'blockdev-del' failed - Node 'drive-scsi0-restore' is busy: node is used as backing hd of '#block655'
After:
> block job (stream) error: restore-scsi0: No space left on device (io-status: ok)
Note that previously, it was not even detected that the stream job
failed and the error message is because the subsequent cleanup failed.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
