When using the path to request properties, and no ZFS file system is mounted
at that path, ZFS will fall back to the parent filesystem:
> # zfs unmount myzpool/subvol-172-disk-0
> # zfs get mounted /myzpool/subvol-172-disk-0
> NAME PROPERTY VALUE SOURCE
> myzpool mounted yes -
> # zfs get mounted myzpool/subvol-172-disk-0
> NAME PROPERTY VALUE SOURCE
> myzpool/subvol-172-disk-0 mounted no -
Thus, we cannot use the path and need to use the dataset directly.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
We allow snapshot names that match pve-configid but during qm destroy we have
not removed all snapshots that match pve-configid so far. For example, the name
x-y was allowed but the resulting snap_vm-105-disk-0_x-y was not removed.
Reported-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
This is basically necessary for the GUI's prune widget, because we want to
pass along all options equal to zero when all the number fields are cleared.
And it's more similar to how it's done in PBS now.
Bumped the APIAGE and APIVER, in case some external plugin needs to adapt to
the now less restrictive schema for 'prune-backups'.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
First, doing such things can make client work slightly easier, as the
submitted values do not need to be made available in any callback
handling the response.
But the actual reason for doing this now is, that this is a
preparatory step for allowing the user to download/print/.. an
autogenerated PBS client encryption key.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
That was lots of code and hash map touching for the case where one
avoided a extra stat, which result probably was in the page cache
anyway, for the case that a backup has a comment.
A case which is rather be unlikely - comments are normally done for
the occasional explicit backup (e.g., before major upgrade, before a
configuration change in that guest, ...), at least not worth some
relatively complicated effort making that sub harder to read and
maintain.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
AFAICT the snapshot activation is not necessary for our plugins at the moment,
but it doesn't really hurt and might be relevant in the future or for external
plugins.
Deactivating volumes is up to the caller, because for example, for replication
on a running guest, we obviously don't want to deactivate volumes.
Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
In order to take a snapshot of a container volume, which can be mounted
read-only with RBD, the volume needs to be frozen (fsfreeze (8)) before taking
the snapshot.
This commit adds helpers to determine if the FIFREEZE ioctl needs to be called
for the volume.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
This replaces a locally maintained hardware map in
get_wear_leveling_info() by commonly used register names of
smartmontool. Smartmontool maintains a labeled register database that
contains a majority of drives (including versions). The current lookup
produces false estimates, this approach hopefully provides more reliable
data.
Signed-off-by: Jan-Jonas Sämann <sprinterfreak@binary-kitchen.de>
Commit 8fe00d9944 already
introduced the necessary logging for the secure code path,
so presumably the bug was already fixed for most people.
Delay the potential die for the send command to be able to log
the ouput+error from the receive command. Like this we also see e.g.
'volume ... already exists' instead of just 'broken pipe'.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
quiet takes care of both the error and success case.
Without this, there are lines like:
myzpool/vm-4352-disk-0@__replicate_4352-7_1601538554__ name myzpool/vm-4352-disk-0@__replicate_4352-7_1601538554__ -
in the log if the dataset exists, and this information is
already present in more readable form.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
we already have the ZFS pool plugin as precedent to use 10s, at for
network with remote off-site storage one can get to 200 - 300ms
RTT latency, which means that for a protocol needing multiple rounds of
communication, one can easily get over 2s while not being in a broken
network.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The LIO backend for ZFS over iSCSI fetches the json-config periodically from
the target.
This patch reduces the stored config values to those which are actually used
and additonally untaints the values read from the remote host's config-file.
Since the LUN index is used in calls to targetcli on the remote host (via
run_command), untainting prevents the call to crash when run with '-T'.
Tested by creating a zfs over iscsi backed VM, starting it, adding disks,
resizing disks, removing disks, creating snapshots, rolling back to a snapshot.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
ZFS over iSCSI fetches information about the disk-images via ssh, thus
the obtainted data is tainted (perlsec (1)).
Since pvedaemon runs with '-T' enabled trying to start a VM via GUI/API failed,
while it still worked via `qm` or `pvesh`.
The issue surfaced after commit cb9db10c1a9855cf40ff13e81f9dd97d6a9b2698 in
pve-common ('run_command: improve performance for logging and long lines'),
and results from concatenating the original (tainted) buffer to a variable,
instead of a captured subgroup.
Untainting the value in ZFSPlugin should not cause any regressiosn, since the
other 3 target providers already have a match on '\d+' for retrieving the
lun number.
reported via pve-user [0].
reproduced and tested by setting up a LIO-target (on top of a virtual PVE),
adding it as storage and trying to start a guest (with a disk on the
ZFS over iSCSI storage) with `perl -T /usr/sbin/qm start $vmid`
[0] https://lists.proxmox.com/pipermail/pve-user/2020-October/172055.html
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
and other stat failure modes.
this method returns undef if 'qemu-img info ...' fails to return
information, so callers must handle this already.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
When creating a new ZFS storage, also instantiate an import-unit for the pool.
This should help mitigate the case where some pools don't get imported during
boot, because they are not listed in an existing zpool.cache file.
This patch needs the corresponding addition of 'zfs-import@.service' in
the zfsonlinux repository.
Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
commit 815df2dd08 introduced a small issue
when activating linked clone volumes - the volname passed contains
basevol/subvol, which needs to be translated to subvol.
using the path method should be a robust way to get the actual path for
activation.
Found and tested by building the package as root (otherwise the zfs
regressiontests are skipped).
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Makes it possible to clone and start a container whose
ZFS subvols are not yet mounted for some reason. If a
subvol cannot be mounted, there's a better error now:
zfs error: cannot mount '/myzpool/subvol-103-disk-0': directory is not empty
Previously, cloning would quietly do an "empty" clone,
and startup would fail with:
mount_autodev: 1074 Permission denied - Failed to create "/dev" directory
lxc_setup: 3238 Failed to mount "/dev"
do_start: 1224 Failed to setup container "103"
__sync_wait: 41 An error occurred in another process (expected sequence number 5)
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
we don't even know whether $snap exists at all, so the old variant could
be rather misleading..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
This is shown in the man page, so it's not important to mention
that this is a wrapper. Also mention the fact that the keep options
from the storage configuration serve as a fallback, which was previously
mentioned in the description of the (now removed) prune-backups parameter.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
For prune selections, it doesn't matter what the current time is,
only the timestamps of the backups matter.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
as else the API cannot easily know if this is set, it cannot check
with -f as the key is in a restricted area and we do not want that a
GET runs as protected.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Implement it for generic storages supporting backups
(i.e. directory-based storages) and add a wrapper for PBS.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Do not try to scan (and thus activate) storages which aren't
configured to support (or cannot support) "vdisks" anyway.
Avoids seemingly strange failures of VM migrations due to a backup storage
not being currently online - even if that storage isn't referenced in
the VM config anywhere..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This is a hack and we should get rid of `run_client_cmd` and
`run_raw_client_cmd` as an API entry!
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
also `pvesm set` and `pvesm add` should behave the same with
respect to how configuration options are treated
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
we already differentiate between standard and non-standard names anyway
and don't detect and return the VMID in the latter case anyway. drop it
from the RE as well to allow names like 'vzdump-qemu-template.vma.lzo'
without the need for a fake VMID.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Because we always have 4-digit years, we can simply pass
the year itself to timelocal instead of subtracting 1900.
Like this it will also work for years not in the range 2000-2999.
See also:
https://perldoc.perl.org/Time/Local.html#Year-Value-Interpretation
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
With Debian Buster it looks like the 'scsi-' method is no longer
reliable. In addition to that, which is also used for non-multipath
systems, add the 'dm-uuid-mpath-' method as fallback. This is also used
by openstack (see os-brick
39b201160b/os_brick/initiator/linuxscsi.py (L400))
Also sort the output of readdir so 'scsi-' is always after
'dm-uuid-mpath-' so the output of pvesm list does not change for systems
that worked before.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
It would be s/bps/pbs/ but as we already have "proxmox-backup-client"
included in the log through the executable name, so it should be
clear that this is a PBS command - so drop that part entirely.
Now using:
> run: /usr/bin/proxmox-backup-client ...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Only expect the logfilename if the archive has a standard name.
This also gives a mechanism to get an untainted filename.
archive_info can take either a volume ID or a path as it's
currently implemented. This is useful for vzdump when there
is no storage (i.e. for 'vzdump --dumpdir'). Add a test case for this.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
This reverts commit 95015dbbf2.
parse_volname always gives 'images' and not 'rootdir'. In most
cases the volume name alone does not contain the needed information,
e.g. vm-123-disk-0 can be both a VM volume or a container volume.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
where 'is_std_name' shows whether the backup name uses the standard naming
schema and most likely was created by our tools.
Also adds a '^' to the existing filename matching regex, which
should be fine since basename() is used beforehand.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
when compiling the disk list add a property with a stable
/dev/disk/by-id/ path for a block device when available.
This is needed to create zpools with the stable by-id links
The /dev/disk/by-id/ directory can contain multiple links to the same device
(e.g. when it's used as a LVM PV, or one for the wwn/nvme-eui in addition
to the one with vendor and serial). We take the first one which matches
the bus where the disk is attached. For nvme disks we exclude the one
containing the nvme-eui.
The patch assumes that not all disks need to have such a link (e.g.
virtio-block devices as we pass them to guests).
Additionally the tests were adapted to run successfully.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
we want to enforce at least the strictness that our tools can do
something with a backup archive..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
* run_command is already imported, use that fact
* avoid useless comments just describing what the code tells one
anyway
* restructure a few parts to more concise/easier to read
implementation.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The rework of the backup file detection logic missed the non-standard
file name case. This patch allows to restore backups with different file
names. Though the config extraction fails, since the type is unknown.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
on an undefined value at /usr/share/perl5/PVE/Storage/Plugin.pm line 928
This error message crops up when a file is deleted after getting the
file list and before the loop passed the file entry.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
since it is a valid content type and adapt the path_to_volume_id_test.
Also adds an extra check if all vtype_subdirs are returned.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
The vzdump file was passed with the full path to the regex. That regex
captures the time from the file name, to calculate the epoch.
As the regex didn't match, the ctime from stat was taken instead. This
resulted in the ctime shown when the file was changed, not when the
backup was made.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Test to reduce the potential for accidental breakage on regex changes.
And to make sure that all vtype_subdirs are parsed.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
with File::stat::stat to minimize variable declarations. And allow to
mock this method in tests instead of the perl build-in stat.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
detection into separate functions so they are reusable and easier
modifiable. This patch also adds the test for archive_info.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
A newer than the Luminous version is shipped with buster, and our
ceph repos are on Nautilus (14.2) in PVE 6.
Allows to drop a check for really old ceph versions (< 10, so
Infernalis and older).
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
dmesg: libceph: bad option at 'conf=/etc/pve/ceph.conf'
After the upgrade to PVE 6 with Ceph Luminous, the mount.ceph helper
doesn't understand the conf= option yet. And the CephFS mount with the
kernel client fails. After upgrading to Ceph Nautilus the option exists
in the mount.ceph helper.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
the '.*' was greedy, also consuming all but one digits of the real percentage
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
switch to \s* instead of .*?, to prevent mis-interpreting potential
strings like '< 50%' or '0-50%'
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
ZFS supports the -p flag in the list command since a few years now.
Let us use the real byte values and avoid the error prone calculation
from human readable numbers that can lead to incorrect numbers if the
reported human readable value is a rounded number.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Getting the volume sizes as byte values instead of converted to human
readable units helps to avoid rounding errors in the further processing
if the volume size is more on the odd side.
The `zfs list` command supports the -p(arseable) flag since a few years
now.
When returning the size in bytes there is no calculation performed and
thus we need to explicitly cast the size to an integer before returning
it.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
to guess a valid volname for a targetstorage of a different type.
This makes it possible to migrate raw volumes between 'dir' and 'lvm'
storages.
It is only used when the storage type for the source storage X
and target storage Y differ and should work as long as Y uses
the standard naming scheme (VMID/vm-VMID-name.fmt respectively vm-VMID-name).
If it doesn't, we get an invalid name and fail, which is the old
behavior (except if X and Y have different types but the same
non-standard naming-scheme, where the old behavior did work).
The original name is preserved, except for a possible extension
and it's also checked if the format is valid for the target storage.
Example: mylvm:vm-123-disk-4 <-> mydir:123/vm-123-disk-4.raw
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
and also return the ID of the allocated volume. This option
allows plugins to choose a new name if there is a collision.
In storage_migrate, the API version of the receiving side is checked.
In Storage.pm's volume_import, when a plugin returns 'undef',
it can be assumed that the import with the requested volid was
successful (it should've died otherwise) and so volid is returned.
This is done for backwards compatibility with foreign plugins.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Instead of relying on list_volumes of Plugin.pm (which filters by
the content types set in the config), use our own to always
show the luns of an iscsi.
This makes sense here, since we need it to show the luns when using
it as base storage for LVM (where we have content type 'none' set).
It does not interfere with the rest of the GUI, since on e.g. disk
creation, we already filter the storages in the dropdown by content
type, iow. an iscsi storage used this way still does not show up
when trying to create a disk.
This also shows the luns now in the 'Content' tab, but this is also
OK, since the user cannot actually do anything there with the luns.
(Besides looking at them)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
With the option valid_target_formats it's possible
to let the caller specify possible formats for the target
of an operation.
[0]: If the option is not set, assume that every format is valid.
In most cases the format of the the target and the format
of the source will agree (and therefore assumption [0] is
not actually assuming very much and ensures backwards
compatability). But when cloning a volume on a storage
using Plugin.pm's implementation (e.g. directory based
storages), the result is always a qcow2 image.
When cloning containers, the new option can be used to detect
that qcow2 is not valid and hence the clone feature is not
available.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
a small grammar fix, and we now return ctime of all files, as
remaining storages are planned for the future omit this hint
completely.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this way the content listing api also returns the vmid on content
listings which, among other things, is useful for the gui for
filtering
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Do not pollute top-level private directory, use "storage" folder but
with backward compatibility.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
1. Avoids the error
qemu-img: The new size must be a multiple of 512
for qcow2 disks.
2. Because volume_import expects disk sizes to be a multiple of 1 KiB.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
As /etc/pve/priv is already pretty polluted, having a
"<storage-id>.pw" file there smells like it could make problems in
the future.
So let the pbs pw file generator use /etc/pve/priv/storages as base
path.
Other storage should move also to that path in the future, if they
save such secrets anywhere in /etc/pve.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
since we redirect the output to our (insecure) socket, logfunc is only
used for STDERR anyway, so we might as well make it explicit on the
caller side.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
