Commit Graph

2784 Commits

Author SHA1 Message Date
Stefan Reiter
0f6c69814b vzdump: log 100% percent in case $target is 0
When $target is 0, that means we don't have to upload any data, in which
case we're immediately done.

Otherwise incremental backups with no changes display a really weird
  status: 0% (0.0 B of 0.0 B), duration 0, read: 0 B/s, write: 0 B/s
when they're actually done already.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-19 18:20:34 +02:00
Stefan Reiter
8825248c43 vzdump: display actually uploaded chunks as 'write' speed
Previously 'read' and 'write' would always show the same value, which is
of little use. Change it so 'write' excludes reused bytes, thus
displaying the actual upload speed.

$last_reused needs to be initialized to contain reused data from 'clean'
dirty bitmaps to ensure the first output line is correct.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-19 18:20:34 +02:00
Stefan Reiter
3a44897f74 vzdump: improve logging output with dirty bitmaps
Uses the new 'query-pbs-bitmap-info' QMP call to retrieve additional
information about each drive's dirty bitmap. Returned info is also used
to calculate $target by simply adding all the dirty values (dirty is
equal to size in case the entire drive will be backed up).

"Backup is sparse" message is suppressed for PBS, since it makes little
sense (if zero chunks appear in the clean area of a bitmap, they won't
be counted, and a user is probably more interested in the 'reused' data
anyway).

Also removes the need for the hacky $first_round query-backup handling.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-19 18:20:34 +02:00
Fabian Ebner
b2e813a6d5 validate_cpu_conf: return config
as parse_property_string expects it to.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Reviewed-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-12 13:54:38 +02:00
Fabian Ebner
a1cbe55cc7 Add missing die statements
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-08-11 14:34:56 +02:00
Fabian Ebner
c15b597137 Add dot character to CPU commandline regular expression
to be able to parse CPU features like "+sse4.2" correctly.

Reported here: https://forum.proxmox.com/threads/zsf-snapshot-unable-to-parse-value-of-runningcpu-got-undefined-value.74094/

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-08-11 12:58:43 +02:00
Fabian Grünbichler
2673ee0b16 d/control: bump versioned dep on pve-storage
needed for PBS plugin methods that are called in backup/restore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-06 09:49:20 +02:00
Fabian Ebner
1264d6c511 Use correct option for storage_migrate
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-08-04 13:57:09 +02:00
Fabian Grünbichler
bd9ab3664d bump version to 6.2-11
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-24 08:17:03 +02:00
Fabian Ebner
aa491a6ecd Fix #2728: die/warn if target is not a replication target when live-migrating
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Tested-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-07-24 07:18:08 +02:00
Fabian Grünbichler
503e96f8de fix #2857: restore: pass keyfile to pbs-restore
if configured. otherwise restoring encrypted backups will be quite
hard..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-20 11:08:26 +02:00
Thomas Lamprecht
6b84c2d66a bump version to 6.2-10
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-13 13:38:42 +02:00
Fabian Ebner
d544e0e0cb set resume parameter for vm_start_nolock if there is a vmstate
In config_to_command, '-loadstate' will be added whenever there is a
vmstate in the config. But in vm_start_nolock, the resume parameter
is used to calculate the appropriate timeout and to remove the vmstate
after the start. The resume parameter was only set if there is a
'suspended' lock, but apparently [0] we cannot rely on the lock to be
set if and only if there is a vmstate.

[0]: https://forum.proxmox.com/threads/task-error-start-failed.72450

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-07-13 12:29:59 +02:00
Thomas Lamprecht
871deaa557 d/control: update build-dependency of pve-doc-generator
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-13 12:25:33 +02:00
Stefan Reiter
5c3b12c7af Add man page cpu-models.conf(5)
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-13 12:24:39 +02:00
Dominik Csapak
7de7f675c2 fix mdev cmdline generation
during refactoring, the vmid got lost, but is necessary to get
the correct mdev id

Fixes commit 74c17b7a23
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ reference fixed commit ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-13 10:29:25 +02:00
Wolfgang Bumiller
d85703a49b pass -X to ssh on upload
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-10 14:24:36 +02:00
Wolfgang Bumiller
70d3ff79a8 bump version to 6.2-9
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-10 14:24:03 +02:00
Wolfgang Bumiller
ece740308f support for encrypted pbs backups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-10 14:23:17 +02:00
Wolfgang Bumiller
dea4b04c1c Add comment about pbs env vars
pbs-restore might not stay there like that forever and if
this code path changes we should remember to also remove the
environment variables

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-10 11:44:54 +02:00
Thomas Lamprecht
1e75990c43 bump version to 6.2-8
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-09 15:14:41 +02:00
Stefan Reiter
b53ba8d0f1 fixup: use parse_property_string instead of parse_cpu_conf_basic
The latter was removed and replaced with a validator.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-09 14:45:21 +02:00
Thomas Lamprecht
b54ba7a170 bump version to 6.2-7
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-09 13:51:25 +02:00
Stefan Reiter
fb9f512c19 backup: detect PBS features and use only supported
If 'query-proxmox-support' is not known to QEMU, assume that no other
features are supported either.

If 'pbs' is not supported at all, error out with a nice message.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-09 13:44:22 +02:00
Fabian Grünbichler
b6f8076eb6 d/control: bump versioned dependency on pve-common
for PVE::JSONSchema with additional validator support

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-08 10:28:20 +02:00
Stefan Reiter
7b8c4de3b8 fix #2671: include CPU format in man page again
Use the new register_format(3) call to use a validator (instead of a
parser) for 'pve-(vm-)?cpu-conf'. This way the $cpu_fmt hash can be used for
generating the documentation, while still applying the same verification
rules as before.

Since the function no longer parses but only verifies, the parsing in
print_cpu_device/get_cpu_options has to go via JSONSchema directly.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-08 10:28:20 +02:00
Thomas Lamprecht
fd98147d6b bump version to 6.2-6
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-07 19:00:20 +02:00
Thomas Lamprecht
91c9f3b062 vzdump: log: small fixups
reuse can also come from the current backup - so drop the "from last
backup" as this can be very confusing if one reads it after making
the first backup ever, with no last backup existing.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-07 17:32:27 +02:00
Thomas Lamprecht
2098f2ff87 vzdump: fix variable redeclaration warning
happened due to moving the code from another scope which had no $res,
and not noticing as it was still working after all.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-07 08:41:30 +02:00
Thomas Lamprecht
277cdcb072 bump version to 6.2-5
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-06 21:59:50 +02:00
Thomas Lamprecht
b4be9c0207 backup: PBS: save backup task size
normally this is done centrally in the managers code, but we do not
have the info for PBS there.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-06 21:59:50 +02:00
Thomas Lamprecht
6cdb568c33 backup: query status: use latest up-to-date info when finished
we used the info from the previous loop for some summary logs, fix
that.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-06 21:59:50 +02:00
Thomas Lamprecht
4d159c24d9 backup: restructure log: use human-readable sizes and include PBS info
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-06 21:59:50 +02:00
Thomas Lamprecht
bbdf1ea5c8 backup: rename confusing incremental param to use-dirty-bitmap
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-06 20:33:13 +02:00
Fabian Ebner
9b29cbd0ed update_disksize: make interface leaner
Pass new size directly, so the function doesn't need to know about
how some hash is organized. And return a message directly, instead
of both size-strings. Also dropped the wantarray, because both
existing callers use the message anyways.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-07-01 09:18:13 +02:00
Fabian Ebner
1c2174833b sync_disks: fix check
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-07-01 09:13:06 +02:00
Thomas Lamprecht
c8239a84dd bump version to 6.2-4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-30 11:34:33 +02:00
Stefan Reiter
69e6289470 enable dirty-bitmap incremental backups for PBS
The $total != $transferred check is changed to a log, as QEMU reports
only actually transferred bytes, and it is indeed correct for
incremental backups to have differing values from $total.

The 'incremental' parameter is always set, QEMU will figure out if it should
re-use an existing bitmap or create a new one on its own.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-29 13:15:47 +02:00
Mira Limbeck
fc701af740 close #2741: add VM.Config.Cloudinit permission
This allows setting ciuser, cipassword and all other cloudinit settings that
are not part of the network without VM.Config.Network permissions.
Keep VM.Config.Network still as fallback so custom roles that add
VM.Config.Network but not VM.Config.Cloudinit don't break.

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2020-06-26 09:41:43 +02:00
Dietmar Maurer
f57666e949 avoid backup command timeout with pbs 2020-06-26 09:30:31 +02:00
Thomas Lamprecht
1fac3a0b31 pci: whitespace, indentation and formating fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-25 13:33:26 +02:00
Stefan Reiter
13d689792e fix #2794: allow legacy IGD passthrough
Legacy IGD passthrough requires address 00:1f.0 to not be assigned to
anything on QEMU startup (currently it's assigned to bridge pci.2).
Changing this in general would break live-migration, so introduce a new
hostpci parameter "legacy-igd", which if set to 1 will move that bridge
to be nested under bridge 1.

This is safe because:
* Bridge 1 is unconditionally created on i440fx, so nesting is ok
* Defaults are not changed, i.e. PCI layout only changes when the new
parameter is specified manually
* hostpci forbids migration anyway

Additionally, the PT device has to be assigned address 00:02.0 in the
guest as well, which is usually used for VGA assignment. Luckily, IGD PT
requires vga=none, so that is not an issue either.

See https://git.qemu.org/?p=qemu.git;a=blob;f=docs/igd-assign.txt

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-25 13:25:35 +02:00
Stefan Reiter
74c17b7a23 cfg2cmd: hostpci: move code to PCI.pm
To avoid further cluttering config_to_command with subsequent changes.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-25 13:25:35 +02:00
Thomas Lamprecht
5a92276e2d trivial whitespace followup fix
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-24 10:59:05 +02:00
Aaron Lauterer
185df962a5 vzdump: move include logic for volumes to method
Move the logic which volumes are included in the backup job to its own
method and adapt the VZDump code accordingly. This makes it possible to
develop other features around backup jobs.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-06-24 10:56:57 +02:00
Fabian Grünbichler
a4e128a9a9 gen_rand_chars: handle errors properly
should not really happen on modern systems, but random_bytes just
returns false if it fails to generate random bytes, in which case we
want to die instead of returning an empty 'random' string.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-06-22 12:03:01 +02:00
Thomas Lamprecht
3c5bdde815 vncproxy: allow to request a generated VNC password
We used the VNC API $ticket as password for VNC, but QEMU limits the
password to the first 8 chars and ignores the rest[0].
As our tickets start with a static string (e.g., "PVE") the entropy
was a bit limited.

For Proxmox VE this does not matters much as the noVNC viewer
provided by has to go always over the API call, and so a valid
ticket and correct permissions for the requested VM are enforced
anyway.

This patch helps external users, which often use NoVNC-Websockify,
circumventing the API and relying solely on the VNC password to avoid
snooping on VNC sessions.

A 'generate-password' parameter is added, if set a password from good
entropy (using libopenssl) is generated.

For simplicity of mapping random bits to ranges we extract 6 bit of
entropy per character and add the integer value of '!' (first
printable ASCII char) to that. This way we get 64^8 possibilities,
which even with millions of guesses per second one would need years
of guessing and mostly just DDOS the server with websocket upgrade
requests.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>

Tested-By: Dominik Csapak <d.csapak@proxmox.com>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
2020-06-22 12:01:55 +02:00
Stefan Reiter
1b7824d349 CPUConfig: add add_cpu_json_properties()
Useful for APIs and docs.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-18 17:11:47 +02:00
Thomas Lamprecht
b63f34b8a1 fix indentation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-18 11:26:36 +02:00
Thomas Lamprecht
326007b250 followup whitespace cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-17 14:05:49 +02:00