similar to the other changes:
- Body to Incoming or proxmox-http's Body
- use adapters between hyper<->tower and hyper<->tokio
- adapt to new proxmox-rest-server interfaces
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
like pbs-client and proxmox-http.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
similar to the http one:
- Body to Incoming for incoming requests
- Body to proxmox-http's Body for everything else
- use legacy client
- use wrappers for hyper<->tower and hyper<->tokio
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
similar changes to proxmox-http:
- Body to Incoming for incoming requests
- Body to proxmox-http's Body for everything else
- switch to "legacy" pooling client from hyper-util
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While eslint is an OK linter, its code formatting capabilities are
rather limited, so replace it with [Biome], which has both a good (and
fast!) linter and code formatter.
[Biome]: https://github.com/biomejs/biome
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
when in blocks, the `var` leaks outside to the function scope, so let us
use `let`
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Use `proxmox-biome format --write`, which is a small wrapper around
[Biome] that sets the desired config options for matching the Proxmox
style guide as close as possible.
Note that the space between function key word and parameter
parenthesis for anonymous functions or function names and parameter
parenthesis for named functions is rather odd and not per our style
guide, but this is not configurable and while it would be relatively
simple to change in the formatter code of biome, we would like to avoid
doing so for both maintenance reason and as this is seemingly the
default in the "web" industry, as prettier–one of the most popular
formatters for web projects–uses this and Biome copied the style
mostly from there.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[TL: add commit message with some background]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Since commit 1e7639bf ("fixup minimum lru capacity") the LRU cache
capacity is set to a minimum value of 1 to avoid issues with the edge
case of 0 capacity.
In commit f1a711c8 ("garbage collection: set phase1 LRU cache
capacity by tuning option") this was not taken into account, allowing
to set values in the range [0, 8*1024*1024] via the datastores tuning
parameters.
Bypass the cache by making it optional and do not use it if the cache
capacity is set to 0, which implies it being disabled.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Since commit 1e7639bf ("fixup minimum lru capacity") the minimum
cache capacity is forced to be 1 to bypass edge cases for it being 0.
Explicitly mention this in the doc comment, as this behavior can be
unexpected.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
`ListGroupsType::new_at` creates a new iterator over all groups of
give backup type with provided parent file descriptor.
The parent directory file descriptor is passed to the `read_subdir`
call, which itself uses it to open the type directory via `openat`.
This call does however ignore the passed file handle if the given
path is absolute [0], which is always the case for the type path
generated via `DataStore::type_path`.
Fix this by passing only the type name as relative path to the
`read_subdir` call, use the absolute path only for
`ListGroupType::new`.
This helps avoiding re-traversing the absolute path in the
`ListGroups` iterator, and since it is then the only callside for
`ListGroupsType::new_at`, inline the instantiation.
[0] https://linux.die.net/man/2/openat
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
`BackupDir::is_protected` is the general helper method to check the
protected state for a snapshot. This checks for the presence of the
protected marker file, which is performed by stating the file and
requires traversing the full path.
When generating the backup list for a backup group, the snapshot
directory contents are however scanned nevertheless. Take advantage
of this by extending the regex used to filter contents by scandir to
include also the protected marker filename and set the state based on
the presence/absence, thereby avoiding the additional stat syscall
altogether.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Explicitly mention that the value sets the available cache slots and
not only mention the value being set to 0 disables the cache, but
rather give also the default and maximum values.
Reported in the community forum:
https://forum.proxmox.com/threads/164869/post-771224
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Removing the group directory when forgetting a backup group or
removing the final backup snapshot of a group did not take into
consideration a potentially present group note file, leading for it
to fail.
Further, since the owner file is removed before trying to remove the
(not empty) group directory, the group will not be usable anymore as
the owner check will fail as well.
To fix this, remove the backup group's note file first, if present
and only after that try to cleanup the rest.
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=6358
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Move and make the helper function to get a backup groups notes file
path a `DataStore` method instead. This allows it to be reused when
access to the notes path is required from the datastore itself.
Further, use the plural `notes` wording also in the helper to be
consistent with the rest of the codebase.
In preparation for correctly removing the notes file from the backup
group on destruction.
No functional changes intended.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
the output of `zpool import` has changed, thus our current parser
failed to find a zpool with zfs userspace in version 2.3.2.
While ZFS 2.3 introduced JSON output for many commands `zpool import`
still lacks the option [0], thus I'd postpone this adapation once all
needed zfs/zpool commands provide JSON.
the change was probably introduced in zfs upstream commit:
5137c132a ("zpool import output is not formated properly.")
[0] https://github.com/openzfs/zfs/issues/17084
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Avoid converting the backup time string to the timestamp and back to
string again. `BackupDir::with_rfc3339` already performs the string
to time conversion, so use it over parsing the timestamp first only
to convert it back to string in `BackupDir::with_group`.
No functional changes intended.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
In case we run into a ready check timeout, query the drive, and
increase the timeout to 2 hours and 5 minutes if it's calibrating (5
minutes headroom). This is effectively a generalization of commit
0b1a30aa ("tape: adapt format_media for LTO9+"), which increased the
timeout for the format procedure, while this here covers also tape
that were not explicitly formatted but get auto-formatted indirectly
on the first action changing a fresh tape, like e.g. barcode labeling.
The actual reason for this is that since LTO-9, initial loading of
tapes into a drive can block up to 2 hours according to the spec. One
can find the IBM and HP LTO SCSI references rather easily [0][1]
As for the timeout, IBM says it only in their recommendations:
> Although most optimizations will complete within 60 minutes some
> optimizations may take up to 2 hours.
And HP states:
> Media initialization adds a variable amount of time to the
> initialization process that typically takes between 20 minutes and
> 2 hours.
So it seems there not a hard limit and depends, but most ordinary
setups should be covered and in my tests it always took around the 1
hour mark.
0: IBM LTO-9 https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference_GA32-0928-05%20(EXTERNAL)_0.pdf
1: HP LTO-9 https://support.hpe.com/hpesc/public/docDisplay?docId=sd00001239en_us&page=GUID-D7147C7F-2016-0901-0921-000000000450.html
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Link: https://lore.proxmox.com/20250415114043.2389789-1-d.csapak@proxmox.com
[TL: extend commit message with info that Dominik provided in a
reply]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
During phase 1 of garbage collection referenced chunks are marked as
in use by iterating over all index files and updating the atime on
the chunks referenced by these.
In an edge case for long running garbage collection jobs, where a
newly added snapshot (created after the start of GC) reused known
chunks from a previous snapshot, but the previous snapshot index
referencing them disappeared before the marking phase could reach
that index (e.g. pruned because only 1 snapshot to be kept by
retention setting), known chunks from that previous index file might
not be marked (given that by none of the other index files it was
marked).
Since commit 74361da8 ("garbage collection: generate index file list
via datastore iterators") this is even less likely as now the
iteration reads also index files added during phase 1, and
therefore either the new or the previous index file will account for
these chunks (the previous backup snapshot can only be pruned after
the new one finished, since locked). There remains however a small
race window between the reading of the snapshots in the backup group
and the reading of the actual index files for marking.
Fix this race by:
1. Checking if the last snapshot of a group disappeared and if so
2. generate the list again, looking for new index files previously
not accounted for
3. To avoid possible endless looping, lock the group if the snapshot
list changed even after the 10th time (which will lead to
concurrent operations to this group failing).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Acked-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Link: https://lore.proxmox.com/20250416105000.270166-3-c.ebner@proxmox.com
Instead of returning a None, fail if the open index reader is called
on a blob file. Blobs cannot be read as index anyways and this allows
to distinguish cases where the index file cannot be read because
vanished.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Link: https://lore.proxmox.com/20250416105000.270166-2-c.ebner@proxmox.com
the below commit accidentally switched this lock to an exclusive lock
when it should just be a shared one as that is sufficient for a
reader:
e2c1866b: datastore/api/backup: prepare for fix of #3935 by adding
lock helpers
this has already caused failed backups for a user with a sync job that
runs while they are trying to create a new backup.
https://forum.proxmox.com/threads/165038
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
Since commit 74361da8 ("garbage collection: generate index file list
via datastore iterators") not only snapshots present at the start of
the garbage collection run are considered for marking, but also newly
added ones. Take these into account by adapting the total index file
counter used for the progress output.
Further, correctly take into account also index files which have been
pruned during GC, therefore present in the list of still to process
index files but never encountered by the datastore iterators. These
would otherwise be interpreted incorrectly as strange paths and logged
accordingly, causing confusion as reported in the community forum [0].
Fixes: https://forum.proxmox.com/threads/164968/
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
else parallel builds of the static binaries will not work correctly, just like
with the regular .do-cargo-build.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
The debian package providing the dynamically linked version of the
proxmox-backup-client is packaged together with the pxar executable.
To be in line and for user convenience, include a statically linked
version of pxar to the static package as well.
Renames STATIC_BIN env variable to STATIC_BINS to reflect that this
now covers multiple binaries and store rustc flags in its own
variable so they can be reused since `cargo rustc` does not allow
invocations with multiple `--package` arguments at once.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Extends the log messages written to the server's backup worker task
log to include the snapshot name which is used as previous snapshot.
This information facilitates debugging efforts, as the previous
snapshot might have been pruned since.
For example, instead of
```
download 'index.json.blob' from previous backup.
register chunks in 'drive-scsi0.img.fidx' from previous backup.
download 'drive-scsi0.img.fidx' from previous backup.
```
this now logs
```
download 'index.json.blob' from previous backup 'vm/101/2025-04-15T09:02:10Z'.
register chunks in 'drive-scsi0.img.fidx' from previous backup 'vm/101/2025-04-15T09:02:10Z'.
download 'drive-scsi0.img.fidx' from previous backup 'vm/101/2025-04-15T09:02:10Z'.
```
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Discurage the use of the statically linked binary for systems where
the regular one is available.
Moves the previous note into it's own section and link to the
installation section.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Link: https://lore.proxmox.com/20250410093059.130504-1-c.ebner@proxmox.com
To have something in the docs.
In the long run we want a somewhat fancy and safe mechanism to host
these builds directly on the CDN and implement querying that for
updates, verified with a backed in public key, but for starters this
very basic docs has to suffice.
We could also describe how to extract the client from the .deb through
`ar` or `dpkg -x`, but that feels a bit to hacky for the docs, maybe
better explained on-demand in the forum or the like.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add a note mentioning that the statically linked binary does not use
the same mechanism for name resolution as the regular client, in
particular that this does not support NSS.
The statically linked binary cannot use the `getaddrinfo` based name
resolution because of possible ABI incompatibility. It therefore is
conditionally compiled and linked using the name resolution provided
by hickory-resolver, part of hickory-dns [0].
[0] https://github.com/hickory-dns/hickory-dns
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
The dependency on the `getaddrinfo` based `GaiResolver` used by
default for the `HttpClient` is not suitable for the statically
linked binary of the `proxmox-backup-client`, because of the
dependency on glibc NSS libraries, as described in glibc's FAQs [0].
As a workaround, conditionally compile the binary using the `hickory-dns`
resolver.
[0] https://sourceware.org/glibc/wiki/FAQ#Even_statically_linked_programs_need_some_shared_libraries_which_is_not_acceptable_for_me.__What_can_I_do.3F
Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
FG: bump proxmox-http dependency
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=4788
Build and package the a statically linked binary version of
proxmox-backup-client to facilitate updates and distribution.
This provides a mechanism to obtain and repackage the client for
external parties and Linux distributions.
The statically linked client is provided as dedicated package,
conflicting with the regular package.
Since the RUSTFLAGS env variables are not preserved when building
with dpkg-buildpackage, invoke via `cargo rustc` instead which allows
to set the recquried arguments.
Credit goes also to Christoph Heiss, as this patch is loosely based
on his pre-existing work for the proxmox-auto-install-assistant [0],
which provided a good template.
Also, place the libsystemd stub into its own subdirectory for cleaner
separation from the compiled artifacts.
[0] https://lore.proxmox.com/pve-devel/20240816161942.2044889-1-c.heiss@proxmox.com/
Suggested-by: Christoph Heiss <c.heiss@proxmox.com>
Originally-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
FG: fold in fixups
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
since it affects whether cargo puts build artifacts directly into
target/debug (or target/release) or into a target-specific
sub-directory.
the package build will always pass `--target $(DEB_HOST_RUST_TYPE)`,
since it invokes the cargo wrapper in /usr/share/cargo/bin/cargo, so
this change unifies the behaviour across plain `make` and `make
deb`.
direct calls to `cargo build/test/..` will still work as before.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This aligns the tooltips to how we have in in Proxmox VE. Using "view"
instead of "open" should make it clear, that this is a safe read-only
action.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Link: https://lore.proxmox.com/20241118104959.95159-1-a.lauterer@proxmox.com
