Commit Graph

522 Commits

Author SHA1 Message Date
Christoph Heiss
a8636bbb66 realm sync: add sync job for AD realms
Basically just a thin wrapper over the existing LDAP-based realm sync
job, which retrieves the appropriate config and sets the correct user
attributes.

Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
2024-04-24 21:06:14 +02:00
Lukas Wagner
a4f1b175d1 server: notifications: send tape notifications via notification system
If the `notification-mode` parameter is set to `legacy-sendmail`, then
we still use the new infrastructure, but don't consider the
notification config and use a hard-coded sendmail endpoint directly.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Lukas Wagner
1d2069d158 server: notifications: send acme notifications via notification system
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Lukas Wagner
823314c7f4 server: notifications: send update notifications via notification system
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Lukas Wagner
5b23a70707 server: notifications: send sync notifications via notification system
If the `notification-mode` parameter is set to `legacy-sendmail`, then
we still use the new infrastructure, but don't consider the
notification config and use a hard-coded sendmail endpoint directly.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Lukas Wagner
2432775ce8 server: notifications: send verify notifications via notification system
If the `notification-mode` parameter is set to `legacy-sendmail`, then
we still use the new infrastructure, but don't consider the
notification config and use a hard-coded sendmail endpoint directly.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Lukas Wagner
3ca03c051f server: notifications: send prune notifications via notification system
If the `notification-mode` parameter is set to `legacy-sendmail`, then
we still use the new infrastructure, but don't consider the
notification config and use a hard-coded sendmail endpoint directly.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Lukas Wagner
04f35d0eee server: notifications: send GC notifications via notification system
If the `notification-mode` parameter is set to `legacy-sendmail`, then
we still use the new infrastructure, but don't consider the
notification config and use a hard-coded sendmail endpoint directly.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Lukas Wagner
57570bda1d notifications: allow sending notifications via proxmox_notify
- Set the context in proxmox_notify
  - Add helper function which queues notifications to a spool
    directory
  - Set up a worker task, running in the privileged process, which
    periodically checks the spool directory for queued notifications

The queuing is needed because on PBS we send most if not all
notifications from the proxy-process running as the `backup` user.
However, to have access to the protected passwords/tokens for various
notification endpoints, we need to read the notification config as
root.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Lukas Wagner
2764be3db5 server: rename email_notifications module to notifications
The module will be extended to interact with the proxmox_notify crate,
hence the name change seems to be in order.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Gabriel Goller <g.goller@proxmox.com>
Reviewed-by: Gabriel Goller <g.goller@proxmox.com>
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 23:14:46 +02:00
Christoph Heiss
132e9722dd realm sync: generic-ify LdapSyncSettings and GeneralSyncSettings
Since both only needs a handful of attributes anyway, pass them
explicitly instead of as an LDAP-specific config object, such that these
types can be reused for other realms like the new Active Directory one.

No functional changes.

Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-25 17:17:12 +01:00
Christian Ebner
ceb639bd0f server: sync job: include removed vanished stats
Include statistics of vanished and therefore removed snapshots, backup
groups and namespaces in the `PullStats`.

In preparation for including these values in the sync jobs task log
output.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-25 17:17:12 +01:00
Christian Ebner
524ed4048b datastore: group: return basic stats on backup group destroy
No functional change intended: In preparation for including the
removed vanished groups and snapshots statistics in a sync jobs task
log output.

Instead of returning a boolean value showing whether all of the
snapshots of the group have been removed, return an instance of
`BackupGroupDeleteStats`, containing the count of deleted and
protected snapshots, the latter not having been removed from the
group.

The `removed_all` method is introduced as replacement for the previous
boolean return value and can be used to check if all snapshots have
been removed. If there are no protected snapshots, the group is
considered to be deleted.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-25 17:17:12 +01:00
Christian Ebner
126322508d server: sync job: format downloaded amount in human readable units
Use the methods provided by HumanByte for the output for consistency
with the rest of the task log and better readability.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-07 14:35:52 +01:00
Christian Ebner
68ac365fea server: sync job: return PullStats for pull related methods
Return basic statistics on pull related methods via `PullStats`
objects, in order to construct a global summary for sync jobs.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Max Carrara <m.carrara@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-03-07 14:34:27 +01:00
Maximiliano Sandoval
c65fee708f report: inline errors in writeln!
Fixes the clippy lint:

```
warning: `to_string` applied to a type that implements `Display` in `writeln!` args
   --> src/server/report.rs:141:72
    |
141 |                 let _ = writeln!(out, "error during read-dir - {}", err.to_string());
    |                                                                        ^^^^^^^^^^^^ help: remove this
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args
```

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-02-13 11:18:50 +01:00
Maximiliano Sandoval
911279b4f9 access first element with first() rather than get(0)
Fixes the clippy lint

```
warning: accessing first element with `self.transports.get(0)`
   --> pbs-tape/src/lib.rs:283:9
    |
283 | /         self.transports
284 | |             .get(0)
    | |___________________^ help: try: `self.transports.first()`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#get_first
    = note: `#[warn(clippy::get_first)]` on by default
```

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-02-13 10:10:19 +01:00
Maximiliano Sandoval
2224b3900d remove needless borrows
Fixes the clippy lint:

```
warning: the borrowed expression implements the required traits
   --> src/server/report.rs:193:47
    |
193 |                         get_directory_content(&path)
    |                                               ^^^^^ help: change this to: `path`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrows_for_generic_args
    = note: `#[warn(clippy::needless_borrows_for_generic_args)]` on by default
```

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-02-13 10:06:16 +01:00
Thomas Lamprecht
c597473080 report: change output contract of functions
let them manage it completely themselves, as we cannot really say if a
code-block fits for the whole output, like it was the case for the
function that returned a limited output of a 'top' process status
command.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-01-18 15:36:32 +01:00
Wolfgang Bumiller
2dd9f98fde cleanup manual unwrap_or_default()
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-01-10 10:13:45 +01:00
Philipp Hufnagl
59c9273698 fix #4315: jobs: modify GroupFilter so include/exclude is tracked
After some discussion I canged the include/exclude behavior to first run
all include filter and after that all exclude filter (rather then
allowing to alternate inbetween). This is done by splitting them into 2
lists, running include first.

A lot of discussion happened how edge cases should be handled and we
came to following conclusion:

no include filter + no exclude filter => include all
some include filter + no exclude filter => filter as always
no include filter +  some exclude filter => include all then exclude

Since a GroupFilter now also features an behavior, the Struct has been
renamed To GroupType (since simply type is a keyword). The new
GroupFilter now has a behaviour as a flag 'is_exclude'.

I considered calling it 'is_include' but a reader later then might not
know what the opposite of 'include' is (do not include?  deactivate?). I
also considered making a new enum 'behaviour' but since there are only 2
values I considered it over engeneered.

Signed-off-by: Philipp Hufnagl <p.hufnagl@proxmox.com>
2024-01-10 10:13:45 +01:00
Stefan Lendl
6b0d7f3d32 report: add prune.cfg to report
prune.cfg stores prune job configurations

Signed-off-by: Stefan Lendl <s.lendl@proxmox.com>
2024-01-09 10:05:26 +01:00
Fabian Grünbichler
4cc4ea64fd sync: fix source store+namespace printing
previously this would always refer to the "top" namespace of the source,
instead of properly iterating over the namespace tree. adapt the trait
accordingly, since this was the only call site.

this fixes a cosmetic issue only.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2023-11-30 11:53:22 +01:00
Fabian Grünbichler
b14e5dcb59 sync: fix recursive remote sync
the snapshot pulling code always selected the "top" namespace of the
source, instead of the passed in namespace parameter.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
2023-11-30 11:52:39 +01:00
Thomas Lamprecht
6685122c3b tree-wide: fix various typos
found with codespell

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-29 18:32:07 +01:00
Lukas Wagner
6ef0238f99 server: sync email notification: adapt template for local sync jobs
Add a guard clause that checks `job.remote`, otherwise the template
fails to render to to handlebars being configured in strict mode.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2023-11-29 17:36:18 +01:00
Thomas Lamprecht
e196f93c3e tree-wide: run cargo fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-29 15:22:43 +01:00
Hannes Laimer
7aeabff238 pull: fix permission checks for local syncs
Reported-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
2023-11-28 17:35:17 +01:00
Thomas Lamprecht
460c3d1619 report: add extra newline between files of directory output
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-28 12:25:01 +01:00
Thomas Lamprecht
5736fa917c report: factor out getting first 30 lines of top output
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-28 12:24:30 +01:00
Hannes Laimer
4ec733279b api: make Remote for SyncJob optional
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Tested-by: Gabriel Goller <g.goller@proxmox.com>
2023-11-25 17:07:42 +01:00
Hannes Laimer
076f36ec4e pull: add support for pulling from local datastore
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Tested-by: Gabriel Goller <g.goller@proxmox.com>
2023-11-25 17:07:42 +01:00
Hannes Laimer
05a52d0106 pull: refactor pulling from a datastore
... making the pull logic independent from the actual source
using two traits.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Tested-by: Gabriel Goller <g.goller@proxmox.com>
2023-11-25 17:07:42 +01:00
Hannes Laimer
09683f1290 accept a ref to a HttpClient
... since the functions don't actually need to own the value.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Tested-by: Gabriel Goller <g.goller@proxmox.com>
2023-11-25 17:07:42 +01:00
Thomas Lamprecht
164f96a5a6 system report: fix warning about checking result
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-11-24 12:02:27 +01:00
Thomas Lamprecht
122179414c system report: record stderr output too
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 18:00:44 +02:00
Thomas Lamprecht
c55884d1a7 system report: factor out getting command output into helper fn
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 17:19:48 +02:00
Thomas Lamprecht
3b5cb8fd33 system report: add proxmox-boot-tool status output
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 17:15:06 +02:00
Thomas Lamprecht
49d465c72a system report: add basic uptime, usage and process info
Command is copied over from Proxmox VE

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 17:14:45 +02:00
Thomas Lamprecht
2f7b3e2ecc system report: add all apt repo files
note, we do not filter by *.list or *.source, so one might get also
files that apt won't read, like .dpkg-dist files, but also those with
typos, and thus possibly helpful when debugging things.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 17:14:45 +02:00
Thomas Lamprecht
0b9614d5a4 system report: support outputting all files in a directory
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 17:14:45 +02:00
Thomas Lamprecht
eb32373e3c system report: group files by categories
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 14:42:41 +02:00
Thomas Lamprecht
20875de2ec system report: switch to markdown-like output
allows one to render this via any of the thousands markdown viewers to
get a better formatting.

We can switch our web ui widget to (optionally) render this as html
when a user is viewing it from the UI too.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 14:42:41 +02:00
Thomas Lamprecht
11c4632bcf system report: output ldap and oidc realm list
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 12:31:44 +02:00
Thomas Lamprecht
84fb190ee2 system report: add more information to lsblk
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 11:32:27 +02:00
Gabriel Goller
48fbce07df fix #4895: scheduled jobs: ignore task-log not found error
Use the job start-time as end-time when it's stuck in the
`JobState::Starting` state, no active working is running and the task
log of the last run doesn't exists.

A user experienced a power loss, which left a GC job in the `Started`
state, but the task log did not exist. This breaks the schedule and no
following GC runs. Now, the error is simply ignored and a new gc job
is started on the next occurrence.

Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
2023-09-27 17:31:22 +02:00
Lukas Wagner
08f8a3e5aa api-types: client: datastore: tools: use proxmox-human-bytes crate
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2023-06-26 13:56:45 +02:00
Maximiliano Sandoval
cd0daa8b33 clippy fixes: the borrowed expression implements the required traits
Suggests that the receiver of the expression borrows the expression.

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2023-06-23 12:01:29 +02:00
Dominik Csapak
bdce7fa154 fix #4719: wait for tape to be available in changer
instead of aborting. If the tape is currently e.g. offline, in an
import/export slot or in the wrong drive, this gives the user the chance to
manually move it/insert it, so that the backup job can continue.

Send an e-mail like we do on a standalone drive, but adapt the messages
to contain the changer instead of the drive.

This can help when not all tapes are currently available in the changer.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2023-05-09 10:23:51 +02:00
Fabian Grünbichler
40a57cfab0 pull: impl Display for SkipReason
instead of manually doing it in SkipInfo's Display implementation.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2023-04-24 15:56:26 +02:00