Commit Graph

43 Commits

Author SHA1 Message Date
Laurențiu Leahu-Vlăducu
922c605d69 rest-server: Improved panic errors with formatted strings
Improved errors when panics occur and the panic message is a
formatted (not static) string. This worked already for &str literals,
but not for Strings.

Downcasting to both &str and String is also done by the Rust Standard
Library in the default panic handler. See:
b605c65b6e/library/std/src/panicking.rs (L777)

Signed-off-by: Laurențiu Leahu-Vlăducu <l.leahu-vladucu@proxmox.com>
2025-01-27 15:12:59 +01:00
Fabian Grünbichler
b2d31f075d rest-server: increase task index lock timeout to 15s
this lock can be quite contended, until the surrounding code is properly split
to reduce this contention it should help to give the worker task
creation/cleanup code a bit more breathing room.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-12-02 16:56:46 +01:00
Fabian Grünbichler
57c8242dc2 rest-server: close race window when updating worker task count
this mimics how the count is updated when spawning a new task - the lock scope
needs to cover the count update itself, else there's a race when multiple
worker's log their result at the same time..

Co-developed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-12-02 16:54:52 +01:00
Fabian Grünbichler
4e51ac3527 rest-server: handle failure in worker task setup correctly
if setting up a new worker fails after it has been inserted into the
WORKER_TASK_LIST, we need to clean it up instead of bubbling up the error right
away, else we "leak" the worker task and it never finishes..

a worker task that never finishes will indefinitely block shutdown
of the rest server process, including the "old" process when reloading
the rest server.

this issue was found in the wild on a system with lock contention on the
file-based lock covering task index updating leading to lock acquiring
timeouts.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-12-02 16:54:18 +01:00
Maximiliano Sandoval
21a39b3cda rest-server: remove use of once_cell
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-08-14 11:32:23 +02:00
Maximiliano Sandoval
25f83bce19 rest-server: remove lazy_static dependency
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
2024-08-14 10:33:42 +02:00
Maximiliano Sandoval
77fe0f6954 docs: clippy: add indentation to doc list items
Fixes the clippy warning:

warning: doc list item missing indentation
   --> proxmox-subscription/src/subscription_info.rs:179:9
    |
179 |     ///  (this mode is used to decide whether to refresh the subscription information)
    |         ^
    |
    = help: if this is supposed to be its own paragraph, add a blank line
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#doc_lazy_continuation
    = note: `#[warn(clippy::doc_lazy_continuation)]` on by default
help: indent this line
    |
179 |     ///   (this mode is used to decide whether to refresh the subscription information)
    |          +

Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
2024-08-07 20:58:04 +02:00
Wolfgang Bumiller
d3abd366c4 introduce proxmox-daemon crate
split from rest-server:
- "state" module (shutdown/reload state)
- shutdown futures
- "daemon" module (named 'server' module in proxmox-daemon)
- command socket

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-24 14:25:50 +02:00
Wolfgang Bumiller
4b9c907b68 log: introduce a shareable LogContext struct
Since hyper can spawn() more tasks, when we stop passing `WorkerTask`
references down the stack, we still need to be able to *inherit* the
current logging context. Hyper provides a way to replace its used
`spawn()` method, so we need to provide a way to reuse the logging
context.

Instead of having the `FileLogger` and warn counter separately
available with local-only access, put them behind an Arc<Mutex<>>.
Previously they already *were* behind an Arc<Mutex<>> as part of the
WorkerTaskState.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-11 14:45:26 +02:00
Wolfgang Bumiller
847a57740b new worker-task crate, move WorkerTaskContext from sys
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-11 14:43:37 +02:00
Gabriel Goller
ddb91a6594 enable tracing logger, remove task_log macros
Enable the tracing-system by setting the LOGGER task local variable
to a instance of a FileLogger and initializing the WARN_COUNTER.
Removed the task_log! macros and some occurences.

Reviewed-by: Lukas Wagner <l.wagner@proxmox.com>
Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
[WB: remove flog! import in doctests]
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-11 14:43:37 +02:00
Wolfgang Bumiller
f1920d9b94 rest-server: fix a build warning
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-11 14:43:37 +02:00
Fabian Grünbichler
6f532dfb7d various clippy fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-06-06 11:40:49 +02:00
Thomas Lamprecht
69edfdf985 rest-server: parse upid: improve on comments and variables
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 10:41:15 +02:00
Thomas Lamprecht
06fed255cb rest-server: extend documentation for reading status of a UPID
Including some possible pitfalls when using this.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 10:26:11 +02:00
Thomas Lamprecht
ba59b84d46 rest-server: code complexity clean-up in task-log rotation
avoid using the negative in the if check and extract the value to be
passed upfront.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 10:13:45 +02:00
Thomas Lamprecht
4197e94e25 rest-server: better document task-log archive rotation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-26 10:13:45 +02:00
Thomas Lamprecht
1e64feeaad rest-server: factor out task-log directory and creation
We had two call sites deriving the directory "shard" where the task
log file is actually saved to, this can lead to ugly bugs and is
better done in a central single-source-of-truth way.

While at it factor out the creation of the log file (and it's shard
directory) to avoid crowding the WorkerTask new fn to much.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2023-10-18 19:36:37 +02:00
Lukas Wagner
de6337ae6d clippy fix: the borrowed expression implements the required traits
See: https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2023-08-08 11:05:20 +02:00
Lukas Wagner
d696ad5bd1 rest-server: add handle_worker from backup debug cli
The function has now multiple users, so it is moved
here.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
2023-02-10 10:23:41 +01:00
Wolfgang Bumiller
5ac4e0fcae more stable clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-08-17 09:22:32 +02:00
Thomas Lamprecht
917f5f73af tree wide: clippy lint fixes
most (not all) where done automatically

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-06-02 15:59:55 +02:00
Thomas Lamprecht
9661defb0f tree wide: some stylistic clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-11 08:14:28 +02:00
Thomas Lamprecht
6b06cc6839 rest server: log rotation: refactor and comment improvements
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-07 14:04:18 +02:00
Thomas Lamprecht
5eaab7864a rest server: log rotation: fix off-by-one for max_days
The entries in a file go from oldest end-time in the first time to
newest end-time in the last line. So, just because the first line is
older than the cut-off time, the remaining one doesn't necessarily
have to be old enough too. What we can know for sure that older than
the current checked rotations of the task archive are definitively up
for deletion.

Another possibility would be to check the last line, but as scanning
backwards is more expensive/complex to do while only being an actual
improvement in a very specific edge case (it's more likely to have a
mixed time-cutoff vs. task-log-file boundary than that those are
aligned)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-07 12:58:32 +02:00
Dominik Csapak
bd8fd62de4 rest-server: add option to rotate task logs by 'max_days' instead of 'max_files'
and use it with the configurable: 'task_log_max_days' of the node config

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:12:49 +02:00
Dominik Csapak
6c856eed5e rest-server: cleanup_old_tasks: improve error handling
by not bubbling up most errors, and continuing on. this avoids that we
stop cleaning up because e.g. one directory was missing.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:10:02 +02:00
Thomas Lamprecht
2bda552b55 rest server: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:55:39 +02:00
Fabian Grünbichler
bb7018e183 misc clippy fixes
the trivial ones ;)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-02-08 14:57:16 +01:00
Fabian Grünbichler
6ad9248cf3 tree-wide: drop redundant clones
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 15:02:07 +01:00
Fabian Grünbichler
647a0db882 tree-wide: fix needless borrows
found and fixed via clippy

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 13:55:33 +01:00
Dietmar Maurer
93625e4f87 update to proxmox-sys 0.2 crate
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
  proxmox-serde instead)

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-24 10:32:27 +01:00
Dietmar Maurer
ef69d1aeb9 use new proxmox-sys crate
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-19 11:06:35 +01:00
Dietmar Maurer
e9bea7b7ed use new fsync parameter to replace_file and atomic_open_or_create
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-21 07:28:32 +02:00
Wolfgang Bumiller
09046671ed update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Dominik Csapak
fb6823b54b rest-server: add cleanup_old_tasks
this is a helper that removes task log files that are not referenced
by the task archive anymore

it gets the oldest task archive file, gets the first endtime (the
oldest) and removes all files in the taskdir where the mtime is older
than that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-08 06:38:52 +02:00
Dietmar Maurer
ad449a5780 rename CommandoSocket to CommandSocket 2021-09-30 12:52:35 +02:00
Dietmar Maurer
9cb2c97c77 proxmox-rest-server: improve docs
And renames abort_worker_async to abort_worker_nowait (avoid confusion,
because the function itself is not async).
2021-09-30 10:51:41 +02:00
Dietmar Maurer
6e50c7aac3 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
Dietmar Maurer
42dae7e1fb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
Dietmar Maurer
020c8e6980 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00
Dietmar Maurer
7a4bb6000e rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer
e8c124fe1b move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00