like we do on a full restore, for that refactor the functionality into
'log_required_tapes' to be reused across the workers
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Specifying the transfer-last parameter limits the amount of backups
that get synced via the pull command/sync job. The parameter specifies
how many of the N latest backups should get pulled/synced. All other
backups will get skipped.
This is particularly useful in situations where the sync target has
less disk space than the source. Syncing all backups from the source
is not possible if there is not enough disk space on the target.
Additionally this can be used for limiting the amount of data
transferred, reducing load on the network.
The newest backup will always get re-synced, regardless of the setting
of the transfer-last parameter.
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
This used to be the case before the switch to the auth api
crate and is required for some helpers where we don't want
to have to setup the complete auth context.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
previously the upid would just be used without a file extension when
downloading a task log. this lead to rather strange filenames that
appeared unfamiliar to users as the upid is not very prevalent in the
gui. set a proper file name based on the node name, worker type and a
time stamp instead. also add the ".log" file extension to indicate
that these files contain logs.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
the backup is finished at that point, the only lock clash that is possible when
dropping the exclusive and attempting to obtain a shared lock would be
- the snapshot is pruned/removed
- the backup is in a pre-upgrade process, and the post-upgrade process opens a reader
the first case is OK, if the other invocation wins the race and removes the
snapshot verification is pointless anyway.
the second case means the snapshot is not verified directly after completion
(this fact would be logged in the backup task log), but usable immediately for
pulling/restoring/..
this should decrease the chances of triggering the issues described in #4523
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the only thing preventing us from using WORM tapes was that we relied
on being able to rewrite the media set label when first using a tape
that was pre-allocated in a media-pool.
so instead of needing to write a meida set label with a special uuid,
just save the pool in the media label itself. This has currently no
downsides, as we're not able to move tapes from one pool to another
anyway.
this makes some checks a bit trickier, as we now have to get the pool
out of the media set label and as a fallback look into the media label.
such new tapes can still be read and restored by older proxmox-bacukp-server
versions. The only thing missing is when a tape labeled with the new
format that has an assigned pool, that pool won't show up when the tape
is inventoried in an old version (but can still be used otherwise).
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since commit 139acf37 ("tape: inventory: skip unassigned tapes")
we skip unassigned tapes (special all-zero media-set uuid) when we look
for a catalog. We accidentally skipped storing it in the inventory
completely, which means we never inventoried completely empty tapes.
to fix that, simply move the check below the inserting in the inventory
Fixes: 139acf37 ("tape: inventory: skip unassigned tapes")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This commits also makes user authentication async, so that e.g. a not
responding LDAP server cannot block other logins.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Note: bind-passwords set via the API are not stored in `domains.cfg`,
but in a separate `ldap_passwords.json` file located in
`/etc/proxmox-backup/`.
Similar to the already existing `shadow.json`, the file is
stored with 0600 permissions and is owned by root.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
instead of
Error: unable to open snapshot directory "/full/path/to/snapshot" for locking - ENOENT: No such file or directory
this will now print
Error: Snapshot vm/800/2023-01-16T12:28:11Z does not exist.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
In former commit, the enum members were renamed to be CamelCase, in
accordance with the usual Rust style guide. However, this broke the
GUI in some places due to failing JSON property deserialization.
To fix this, some serde(rename = "kebab-case") directives were added.
Some properties were also serde-renamed to snake_case, otherwise
it would have been necessary to also modify proxmox-widget-toolkit
as well as PVE source code. This can follow in a later commit if so
desired.
Fixes: a2055c38 fix non-camel-case enums
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
zfs errors might include a newline in the output (e.g. when trying to
create a mirror on two differently sized disks), which trips up our
task log status parser since that expectes a 'TASK {status}' on the
beginning of the first line.
print the error from zfs into the log and bail out with a short notice
to check the task log
this fixes the 'unknown error' issue in the ui when an error happnes
during the zfs commands
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
.. in the same way the PVE api does, esp. regarding the logic to handle
oneshot and missing services.
This then allows re-using the GUI parts from there as well, so that the
services page in PVE and PBS looks the same.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
tapes that are labeled into a pool but are not in a media-set yet, belong
to the special 'all zero' media-set. these will never have a catalog on them,
so skip them
fixes the issue, that an inventory with 'catalog restore' aborted on
such a tape
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
a tape assigned to a pool but no media-set, gets the special 'all zero'
media set in it's MediaSetLabel. Instead of having that constant
scattered all over the code, hide this fact by using wrapper functions
to initialize it that way and to check for it
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The API now exposes the field 'available' as well, with which the
unprivileged total is calculated in all corresponsing views in the
frontend.
The rrd charts now also display the total as the unprivileged total
if available, otherwise the absolute total is used.
Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
The rrd data now includes tracking the available field in disk usage.
The calculation for the estimated_time_full was adapted to use the
total for the unpriviliged user, which is the sum of used + available.
The total for unprivileged users is preferable, because datastores are
always written to by the backup user. Which means that any storage
space reserved for root is unusable for our purposes.
To avoid resetting the estimate when switching to this new version,
the backend will try to use the available value to calculate the
unprivileged total. When that is not an option, it will fall back to
using the absolute total.
Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
The read_tasklog API call now stream the whole log file if the query
parameter 'download' is set to true. If the limit parameter is set to
0, all lines in the tasklog will be returned in json format.
To make a file stream and a json response in the same API call work, I
had to use one of the lower level apimethod types from the
proxmox-router. Therefore, the routing declarations and parameter
schemas have been changed accordingly.
Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
currently, we don't (f)sync on chunk insertion (or at any point after
that), which can lead to broken chunks in case of e.g. an unexpected
powerloss. To fix that, offer a tuning option for datastores that
controls the level of syncs it does:
* None (default): same as current state, no (f)syncs done at any point
* Filesystem: at the end of a backup, the datastore issues
a syncfs(2) to the filesystem of the datastore
* File: issues an fsync on each chunk as they get inserted
(using our 'replace_file' helper) and a fsync on the directory handle
a small benchmark showed the following (times in mm:ss):
setup: virtual pbs, 4 cores, 8GiB memory, ext4 on spinner
size none filesystem file
2GiB (fits in ram) 00:13 0:41 01:00
33GiB 05:21 05:31 13:45
so if the backup fits in memory, there is a large difference between all
of the modes (expected), but as soon as it exceeds the memory size,
the difference between not syncing and syncing the fs at the end becomes
much smaller.
i also tested on an nvme, but there the syncs basically made no difference
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in a disaster recovery case, it is useful to not only re-invetorize
the labels + media-sets, but also to try to recover the catalogs
from the tape (to know whats on there). This adds an option to
the inventory api call that tries to do a fast catalog restore
from each tape to be inventorized.
also sets the correct default for 'read-all-labels' in the api and
converts to a bool
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this way we can omit the pattern
```
let status_path = Path::new(TAPE_STATUS_DIR);
some_function(status_path);
```
and give the TAPE_STATUS_DIR directly. In some instances we now have to
give TAPE_STATUS_DIR more often, but most often we save a few
intermediary Paths.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we converted the prune settings of datastores to prune-jobs, but did
not actually implement the notifications for them, even though
we had the notification options in the gui (they did not work).
implement the basic ok/error notification for prune jobs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Some of them could easily be grouped in a kind of
RestoreWorker struct, but that'll still leave one bigger
function that's more annoying to change.
Let's just allow it for now.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
