Commit Graph

254 Commits

Author SHA1 Message Date
Thomas Lamprecht
345243fa7e tree-wide: fix typos with codespell
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit ffc31266da)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-23 08:41:05 +02:00
Dominic Jäger
4b84ad5e25 Optionally allow blockdev in abs_filesystem_path
This is required to import from LVM storages, for example

Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
2021-03-31 11:00:25 +02:00
Fabian Ebner
2c5246e1ea vdisk list: allow specifying content type
and only scan storages that support it if specified.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-31 10:22:52 +02:00
Fabian Ebner
26d7314be0 Revert "vdisk list: only collect images from storages with an appropriate content type"
This reverts commit a44c18925d and adds a reminder
comment.

The mentioned commit is actually a backwards-incompatible change that leads to
slightly different behavior when migrating a VM with volumes on a misconfigured
storage. For example, unreferenced volumes on a misconfigured storage won't be
picked up, even though they were before. And for referenced volumes on a
misconfigured storage, the disk size would not be updated on migration anymore.

We should wait until the next major release for this change and then also
re-evaluate the migration behavior with misconfigured disks.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-31 10:22:52 +02:00
Fabian Ebner
a44c18925d vdisk list: only collect images from storages with an appropriate content type
Only these storages are activated in the first place, and it's bad behavior to
list images when no appropriate content type is not set.

For example, on VM destruction, this avoids unreferenced images to be deleted
from a storage with only 'backup' content type set, which is supposedly what
happened in this[0] forum thread.

(Some) callers expect all keys to be present and valid array references in the
result, so initialization is needed.

Now, the enabled check is already done by the preceding code for every element
that is iterated over, and thus isn't needed in the main loop anymore.

[0]: https://forum.proxmox.com/threads/erasing-all-vm-disks-after-a-failed-vm-migration-task.85068

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-03-15 13:36:53 +01:00
Fabian Ebner
7cdc75a290 storage migration: insecure: improve logging
by including the message/error from the remote side. Some people on the forum[0]
ran into 'no tunnel IP received', but without information from the remote side
it's hard to tell why.

[0]: https://forum.proxmox.com/threads/failed-no-tunnel-ip-received.80172

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-19 15:29:04 +01:00
Fabian Ebner
50853be2c5 remove lock from is_base_and_used check
and squash the __no_lock-variant into it.

This lock is not broad enough, because for a caller that plans to do or not do
some storage operation based on the result of the check, the following could
happen:
1. volume_is_base_and_used is called and the result is used to enter a branch
2. situation on the storage changes in the meantime
3. the branch chosen in 1. might not be the one that should be taken anymore

This means that callers are responsible for locking, and luckily the existing
callers do use their own locks already:
1. vdisk_free used the __no_lock-variant with a broader lock also covering
   the free operation.
2. vdisk_clone is not a caller, but is relevant and it does lock the storage
2. the calls during VM migration and VM destruction happen in the context of a
   locked VM config. Because the clone operation also locks the VM config, it
   cannot happen that a linked clone is created while the template VM is
   migrated away or destroyed or vice versa. And even if that were the case,
   the base disk would not be freed, because of what vdisk_free/vdisk_clone do.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-02-06 14:47:40 +01:00
Fabian Grünbichler
d3a5e30963 drop absolute udevadm path
the compat symlink from bin to sbin has been dropped with bullseye, and
we rely on PATH begin set properly in our daemons/CLI tools anyway..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 18:27:14 +01:00
Fabian Ebner
189e67ffa6 fix #3199: by fixing usage of strftime
In a very early version I wanted to parse the date from the backup
name, and when switching to using the ctime and localtime() instead,
I forgot to update the usage of strftime.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-12-15 14:40:45 +01:00
Fabian Ebner
10dfeb9ef0 prune mark: correctly keep track of already included backups
This needs to happen in a separate loop, because some time intervals are not
subsets of others, i.e. weeks and months. Previously, with a daily backup
schedule, having:
* a backup on Sun, 06 Dec 2020 kept by keep-daily
* a backup on Sun, 29 Nov 2020 kept by keep-weekly
would lead to the backup on Mon, 30 Nov 2020 to be selected for keep-monthly,
because the iteration did not yet reach the backup on Sun, 29 Nov 2020 that
would mark November as being covered.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-12-14 16:11:29 +01:00
Dominik Csapak
e9991d2694 Storage/Plugin: add get/update_volume_comment and implement for dir
and add the appropriate api call to set and get the comment
we need to bump APIVER for this and can bump APIAGE, since
we only use it at this new call that can work with the default
implementation

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-24 10:23:25 +01:00
Fabian Ebner
d4e8a1f27c convert maxfiles to prune_backups when reading the storage configuration
If there are already prune options configured, simply delete the maxfiles
setting. Having set both is invalid from vzdump's perspective anyways, and any
backup job on such a storage failed, meaning a user would've noticed.

If there are no prune options, translate the maxfiles value to keep-last,
except for maxfiles being zero (=unlimited), in which case we use keep-all.

If both are not set, don't set anything, so:
1. Storages don't suddenly have retention options set.
2. People relying on vzdump defaults can still use those.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-23 15:57:30 +01:00
Fabian Ebner
1b87f01388 prune: introduce keep-all option
useful to have an alternative to the old maxfiles = 0. There has to
be a way for vzdump to distinguish between:
1. use the /etc/vzdump.conf default (when no options are configured for the storage)
2. use no limit (when keep-all=1)

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-23 15:27:17 +01:00
Thomas Lamprecht
1207620cf0 perlcritic: avoid conditional variable declaration
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-23 15:24:02 +01:00
Fabian Ebner
343ca2570c prune: allow having all prune options zero/missing
This is basically necessary for the GUI's prune widget, because we want to
pass along all options equal to zero when all the number fields are cleared.
And it's more similar to how it's done in PBS now.

Bumped the APIAGE and APIVER, in case some external plugin needs to adapt to
the now less restrictive schema for 'prune-backups'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-16 10:14:11 +01:00
Fabian Ebner
f514181d28 prune mark: keep all if all prune options are zero/missing
as an additional safety measure. And add some tests.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-16 10:14:11 +01:00
Fabian Ebner
5324ceffef fix #3030: always activate volumes in storage_migrate
AFAICT the snapshot activation is not necessary for our plugins at the moment,
but it doesn't really hurt and might be relevant in the future or for external
plugins.

Deactivating volumes is up to the caller, because for example, for replication
on a running guest, we obviously don't want to deactivate volumes.

Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-10 19:04:58 +01:00
Stoiko Ivanov
2c036838ed add check for fsfreeze before snapshot
In order to take a snapshot of a container volume, which can be mounted
read-only with RBD, the volume needs to be frozen (fsfreeze (8)) before taking
the snapshot.

This commit adds helpers to determine if the FIFREEZE ioctl needs to be called
for the volume.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-11-10 18:58:45 +01:00
Stoiko Ivanov
af2dd59eaf fix typo in comment
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-11-10 18:58:45 +01:00
Fabian Ebner
57acd6a124 fix #1452: also log stderr of remote command with insecure storage migration
Commit 8fe00d9944 already
introduced the necessary logging for the secure code path,
so presumably the bug was already fixed for most people.

Delay the potential die for the send command to be able to log
the ouput+error from the receive command. Like this we also see e.g.
'volume ... already exists' instead of just 'broken pipe'.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-10-28 14:05:49 +01:00
Fabian Ebner
93d1812e5a storage_migrate: log bandwidth limit
and avoid undefined post-if declaration of @cstream.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-08-04 13:57:24 +02:00
Fabian Ebner
8f26b3910d Add prune_backups to storage API
Implement it for generic storages supporting backups
(i.e. directory-based storages) and add a wrapper for PBS.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-07-24 15:44:53 +02:00
Thomas Lamprecht
c43655d2ed vdisk_list: skip scanning storages which cannot have images/rootdisks
Do not try to scan (and thus activate) storages which aren't
configured to support (or cannot support) "vdisks" anyway.

Avoids seemingly strange failures of VM migrations due to a backup storage
not being currently online - even if that storage isn't referenced in
the VM config anywhere..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-09 16:16:23 +02:00
Fabian Grünbichler
28be2a431b archive_info: relax custom name handling
we already differentiate between standard and non-standard names anyway
and don't detect and return the VMID in the latter case anyway. drop it
from the RE as well to allow names like 'vzdump-qemu-template.vma.lzo'
without the need for a fake VMID.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-08 15:26:19 +02:00
Fabian Ebner
b1ddc54a93 archive_info: use timelocal correctly
Because we always have 4-digit years, we can simply pass
the year itself to timelocal instead of subtracting 1900.
Like this it will also work for years not in the range 2000-2999.

See also:
https://perldoc.perl.org/Time/Local.html#Year-Value-Interpretation

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-07-08 10:45:49 +02:00
Thomas Lamprecht
d37729b95b scan_cifs: fix scanning server with no SMB1 fallback
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-01 15:46:30 +02:00
Thomas Lamprecht
8594155a66 scan cifs: fix passing "no pass" parameter
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-01 12:17:02 +02:00
Thomas Lamprecht
c42d86559c scan_cifs: do not add NT_STATUS lines to result
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-01 12:11:59 +02:00
Thomas Lamprecht
67a2371180 scan_cifs: do not enforce password for users
there can be accounts with explicit null password others than the
mapped guest account.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-01 12:11:59 +02:00
Thomas Lamprecht
afaa98f10f scan_cifs: pass user/pass over environment
As command line argument they are readable by ever user in the same
PID namespace.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-01 12:06:53 +02:00
Thomas Lamprecht
fd55f51efa scan_cifs: add workgroup domain independent of user param
they are not really bound to each other

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-01 12:06:53 +02:00
Thomas Lamprecht
be18d6dac7 scan_cifs: raise smbclient errors
Silencing error messages can extend the search for an issue while
having to advantage.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-01 12:06:53 +02:00
Thomas Lamprecht
436773fd78 scan_cifs: fix comment typo and indentation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-01 12:06:53 +02:00
Thomas Lamprecht
9afd3be18c add trailing newline to error message
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-30 14:19:24 +02:00
Fabian Ebner
35a3953213 Add archive_remove
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-06-30 13:59:23 +02:00
Fabian Ebner
e34afeb117 Extend archive_info to include filename and logfilename
Only expect the logfilename if the archive has a standard name.
This also gives a mechanism to get an untainted filename.

archive_info can take either a volume ID or a path as it's
currently implemented. This is useful for vzdump when there
is no storage (i.e. for 'vzdump --dumpdir'). Add a test case for this.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-06-30 13:59:23 +02:00
Fabian Ebner
12442ae19c Fix 2763: Revert "storage_migrate: check if target storage supports content type"
This reverts commit 95015dbbf2.

parse_volname always gives 'images' and not 'rootdir'. In most
cases the volume name alone does not contain the needed information,
e.g. vm-123-disk-0 can be both a VM volume or a container volume.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-06-23 14:43:17 +02:00
Fabian Ebner
fb821c1828 Expand archive_info to include ctime, vmid and is_std_name
where 'is_std_name' shows whether the backup name uses the standard naming
schema and most likely was created by our tools.

Also adds a '^' to the existing filename matching regex, which
should be fine since basename() is used beforehand.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-06-06 19:38:02 +02:00
Thomas Lamprecht
5029f978c3 archive info: keep some basic strictness
we want to enforce at least the strictness that our tools can do
something with a backup archive..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-15 19:18:24 +02:00
Thomas Lamprecht
bf5af0fbf5 archive info: include archive name in error message
so that we have some context if users report issues with it..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-15 19:13:39 +02:00
Thomas Lamprecht
63e892950b storage: archive/extract code cleanup
* run_command is already imported, use that fact
* avoid useless comments just describing what the code tells one
  anyway
* restructure a few parts to more concise/easier to read
  implementation.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-15 18:46:17 +02:00
Alwin Antreich
4b26f8140d Fix: backup: relax file name matching regex
The rework of the backup file detection logic missed the non-standard
file name case. This patch allows to restore backups with different file
names. Though the config extraction fails, since the type is unknown.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-05-15 18:12:01 +02:00
Alwin Antreich
014d36dbbb Fix: #2124 storage: add zstd support
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
Alwin Antreich
277cafc0ff backup: compact regex for backup file filter
the more compact form of the regex should allow easier addition of new
file extensions.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
Alwin Antreich
73fcb7bfd1 Fix: add missing snippets subdir
since it is a valid content type and adapt the path_to_volume_id_test.
Also adds an extra check if all vtype_subdirs are returned.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
Alwin Antreich
892dc99249 Fix: path_to_volume_id returned wrong content
type for backup files. Patch includes changes of the test as well.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
Alwin Antreich
cd554b79d1 storage: test: split archive format/compressor
detection into separate functions so they are reusable and easier
modifiable. This patch also adds the test for archive_info.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2020-04-30 18:37:19 +02:00
Fabian Ebner
2668adcec2 Add comment for volume_has_feature
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-09 12:05:11 +02:00
Aaron Lauterer
3881e68025 ZFS: use -p flag and remove zfs_parse_size
ZFS supports the -p flag in the list command since a few years now.
Let us use the real byte values and avoid the error prone calculation
from human readable numbers that can lead to incorrect numbers if the
reported human readable value is a rounded number.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-04-09 10:20:06 +02:00
Fabian Ebner
683a3f463f storage_migrate: add volname_for_storage helper
to guess a valid volname for a targetstorage of a different type.
This makes it possible to migrate raw volumes between 'dir' and 'lvm'
storages.

It is only used when the storage type for the source storage X
and target storage Y differ and should work as long as Y uses
the standard naming scheme (VMID/vm-VMID-name.fmt respectively vm-VMID-name).
If it doesn't, we get an invalid name and fail, which is the old
behavior (except if X and Y have different types but the same
non-standard naming-scheme, where the old behavior did work).

The original name is preserved, except for a possible extension
and it's also checked if the format is valid for the target storage.
Example: mylvm:vm-123-disk-4 <-> mydir:123/vm-123-disk-4.raw

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-04-09 09:41:01 +02:00