The protected status of the snapshot is retrieved twice. This is slow
because it stat's the .protected file multiple times.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
by skipping the payloader reader entirely, it's not needed for listing contents
and would make accessing larger archives too expensive.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Allow to pass the archive name as optional api call parameter instead
of having it as prefix to the path.
If this parameter is given, instead of splitting of the archive name
from the path, the parameter itself is used, leaving the path
untouched.
This allows to restore single files from the archive, without having
to artificially construct the path in case of file restores for split
pxar archives, where the response path of the listing does not
include the archive, as opposed to the response provided by lookup
via the catalog.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Add an optional `archive-name` parameter, indicating the metadata
archive to be used for directory content lookups instead of the
catalog. If provided, instead of the catalog reader, a pxar Accessor
instance is created to perform the lookup.
This is in preparation for dropping catalog encoding for snapshots
with split pxar archive encoding.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Move code that can be reused when having to perform a lookup via the
pxar metadata archive instead of the catalog out of the thread.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
The file path passed to the catalog is base64 encoded, with an exception
for the root.
Factor this check and decoding step out into a helper function to make
it reusable when doing the same for lookups via the metadata archive
instead of the catalog.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Attach the payload chunk reader for pxar archives which have been
uploaded using split streams for metadata and payload data.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
... and adapt to the new reader/writer variant for encoder or
decoder/accessor to attach a dedicated payload input/output for split
pxar archives.
In preparation for look-ahead caching, where a passing around of
per-directory level encoder instances with internal references is
not feasible.
Previously, for each directory level a new encoder instance has been
generated, restricting possible implementation errors. These encoder
instances have been internally linked by references to keep track of
the state changes in a parent child relationship.
This is however not feasible when the encoder has to be passed by
mutable reference, as required by the look-ahead cache
implementation. The encoder has therefore been adapted to use a
single instance implementation with an internal stack keeping track
of the state.
Depends on the bumped pxar library version, including the patches to
attach the corresponding variant for the pxar reader/writer
instantiation.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Move the code to get the local chunk reader to a dedicated function
to make it reusable. The same code is required to get the local chunk
reader for the payload stream for split stream archives.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
the schedule handling is the same whether there was a last run or not, so let's
do it once and not twice. the duration can be stored right away, instead of
using an intermediate variable.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the latter was newly introduced, and they both return basically the same
information now. the new extended (job) status struct is a strict superset of
the old status struct, so this is not a breaking change API wise.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
to avoid drifting definitions and reduce duplication. with the next major
release, the 'upid' field could then be renamed and aliased to be in line with
the other jobs, which all use 'last-run-upid'. doing it now would break
existing callers of the GC status endpoint (or consumers of the on-disk status
file).
the main difference is that the GC status fields are now not optional (except
for the UPID) in the job status, since flattening an optional value is not
possible. this only affects datastores that were never GCed at all, and only
direct API consumers, since the UI handles those fields correctly.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Adds an api endpoint on the datastore that reports the gc job status
such as:
- Schedule
- State (of last run)
- Duration (of last run)
- Last Run
- Next Run (if scheduled)
- Pending Chunks (of last run)
- Pending Bytes (of last run)
- Removed Chunks (of last run)
- Removed Bytes (of last run)
Adds a dedicated endpoint admin/gc that reports gc job status for all
datastores including the onces without a gc-schedule.
Signed-off-by: Stefan Lendl <s.lendl@proxmox.com>
Originally-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewd-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
`prune-group` is currently not a real workertask, ie it behaves like one
but doesn't start a thread nor a task to do its work.
Changed it to start a tokio-task, so that we can delete snapshots
asynchronously. The `dry-run` feature still behaves in the same way and
returns early.
This paves the way for the new logging infra (which uses `task_local` to
define a logger) and improves performance of bigger backup-groups.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
No functional change intended: In preparation for including the
removed vanished groups and snapshots statistics in a sync jobs task
log output.
Instead of returning a boolean value showing whether all of the
snapshots of the group have been removed, return an instance of
`BackupGroupDeleteStats`, containing the count of deleted and
protected snapshots, the latter not having been removed from the
group.
The `removed_all` method is introduced as replacement for the previous
boolean return value and can be used to check if all snapshots have
been removed. If there are no protected snapshots, the group is
considered to be deleted.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The API now exposes the field 'available' as well, with which the
unprivileged total is calculated in all corresponsing views in the
frontend.
The rrd charts now also display the total as the unprivileged total
if available, otherwise the absolute total is used.
Signed-off-by: Daniel Tschlatscher <d.tschlatscher@proxmox.com>
In the auth code we rather #[allow] the binding, because in
this case we explicitly want to assert the type.
In fact, it would make more sense for clippy to not warn
about a unit type if the unit type is explicitly spelled
out.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
the remaining ones are:
- type complexity
- fns with many arguments
- new() without default()
- false positives for redundant closures (where closure returns a static
value)
- expected vs actual length check without match/cmp
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
This is a stop-gap measure to prevent snapshot listing from
blocking the main async worker threads as it can potentially
do a *lot* of I/O.
Ideally we'll move to a proper streaming API, but this will
be an API break.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Use the moved 'fs_info' helpers from the proxmox-sys crate (available
from there since proxmox-sys 0.3.0) as replacement for 'disk_usage'
in the workspace local tools crate and remove the latter as we do not
need it anymore.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ T: squashed in removal of now unused import and reworded commit
message to include version availability info, among other things ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Make the assumption that if a user has any privilege that would make
an NS and (parts) of its content visible they also should be able to
know about the datastore and very basic errors on lookup (path
existence and maintenance mode) even if that NS doesn't even exists
(yet), as they could, e.g., make or view a backup and find out
anyway.
This avoids iterating over parts of the whole datastore folder tree
on disk, doing a priv check on each, swapping IO to virtual in memory
checks on info we got available already anyway, is always a good idea
after all
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to avoid the problematic open fresh datastore with fresh chunkstore
with, and that's the actual problematic part, fresh process locker.
As the latter uses posix record locks which are pretty dangreous as
they operate on a path level (not FD level) and thus closing any file
opened (even if it wasn't opened for locking at all) drops all active
locks on the same file on completely unrelated file descriptors -.-
Also, no operation wasn't exactly correct for this thing in the first
place, but we cannot use Operation::Lookup either, as we're currently
indeed using a rather stupid-simple way and *are* reading.
So until we optimize this to allow querying the AclTree if there's
any priv XYZ below a path, use the Operation::Read.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
I.e., for those that only got permissions on a sub namespace and
those that onlöy got BACKUP_READ, as both they could just list and
count themselves too after all, so not exactly secret info.
The UI needs some adaptions to cope with gc-stats and usage being
optional, will be done in a next commit.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead move the acl_path helper to BackupNamespace, and introduce a new
helper for printing a store+ns when logging/generating error messages.
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
