Commit Graph

1558 Commits

Author SHA1 Message Date
Max Carrara
090497ec08 section config: update code style
Replace `foreach` with `for` and use postfix deref instead of block
(circumfix) dereference (`$foo->%*` instead of `%$foo`).

Furthermore, make `format_config_line` a private sub instead of
unnecessarily declaring it as an anonymous subroutine, which avoids
the `&$sub_ref(...)` syntax altogether.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-11-11 14:01:02 +01:00
Max Carrara
142fb2b245 section config: document package and its methods with POD
Apart from the obvious benefits that documentation has, this also
allows LSPs to provide docstrings e.g. via 'textDocument/hover' [0].

Tested with Perl Navigator [1].

[0]: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_hover
[1]: https://github.com/bscan/PerlNavigator

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-11-11 14:01:02 +01:00
Thomas Lamprecht
15075a54e9 sysfs tools: avoid using POSIX for just EEXIST error code
We can just save both $! and %! and use the latter to check for
specific errors. This is not really pretty but perl does the same
internally, so...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-11-11 13:52:43 +01:00
Dominik Csapak
3b6ce501ad sysfstools: fix regression on binding to vfio-pci
when starting a vm with passthrough, we have to bind all normal pci
devices to vfio-pci. This happens by

* unbinding from current driver
* telling vfio-pci the 'vendorid modelid' combo so it knows this device
  class can use the driver (by writing to 'new_id')
* actually binding the device to vfio-pci

if there are multiple devices of the same 'vendorid modelid' class on
the host (and passed through), only the first write to 'new_id' is
successful, all subsequent ones return EEXIST.

This could happen e.g. for setups with multiple GPUs that have the same
audio chip.

To fix this, ignore the EEXIST error for this write to new_id, by adding
a new parameter to file_write for this.

If we need to ignore other errors in the future, we can still make this
more generic.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-11-11 13:38:45 +01:00
Dominik Csapak
e8ab2427d6 sysfstools: file_write: extend with logging
the actual error and path is useful to know when trying to debug or
figure out what did not work, so warn here if there was an error.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-11-11 13:38:42 +01:00
Fiona Ebner
bbca06d995 inotify: avoid cyclic use statement
Commit e68ebda ("fix #545: interfaces: allow arbitrary bridge names in
network config") introduced a cyclic usage between
PVE::RESTEnvironment and PVE::INotify, making code like the following
fail:

> perl -e "use PVE::RESTEnvironment qw(log_warn);"

Note, including the PVE::INotify module first would still work, i.e.:

> perl -e "use PVE::INotify; use PVE::RESTEnvironment qw(log_warn);"

The rest of the PVE::INotify module alredy uses syslog(), which could
be used here as well to get rid of the cyclic usage. Wolfgang argued
that the whole point of commit e68ebda was to remove coupling between
the name and the type of the interface. If there still is some code
about a name starting with 'vmbr' being classified wrong, that should
rather be fixed. Because of the very commit, the frontend already
doesn't show e.g. a non-bridge with name 'vmbr7' in bridge selectors.

Suggested-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: e68ebda ("fix #545: interfaces: allow arbitrary bridge names in network config")
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-11-11 13:37:01 +01:00
Thomas Lamprecht
61073db2bc certificate: factor out obtaining the expiration timestamp
This can be useful to have, e.g., when requiring different behaviors
the nearer an expiry gets.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-11-11 09:51:59 +01:00
Tiomet Pelston
8915b9669c fix #5623: ovs other_config set to 0 not saved in network config
When configuring an OVS network device via web interface,
any OVS option set to value=0 is ignored upon saving. This happens
because value=0 is evaluated as false in $parse_ovs_option.

Signed-off-by: Tiomet Pelston <tiometpelston@gmail.com>
Reviewed-By: Aaron Lauterer <a.lauterer@proxmox.com>
Tested-By: Aaron Lauterer <a.lauterer@proxmox.com>
2024-10-30 09:51:34 +01:00
Thomas Lamprecht
0b1b9f05ca bump version to 8.2.6
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-24 18:37:49 +02:00
Dominik Csapak
2d658bfeca SysFSTools: lscpi: move mdev and iommugroup check outside of verbose
they should not be expensive (only reading/file checking in sysfs; the
parsed vendor/id names are not required) so we should include them
always.

We need at least the mdev part later at a point where we're not
interested in the rest of the verbose mode.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>
2024-10-24 18:33:51 +02:00
Dominik Csapak
dc023d60c5 SysFSTools: handle new nvidia syfsapi as mdev
with kernel 6.8 NVIDIAs vGPU driver has a different api than the
previous 'mediated devices'. Adapt our sysfcode to also recognize this
for the 'mdev' paths and add another 'nvidia' property so we can detect
this.

Also parse the new api when they exist instead of the mediated devices.

The biggest difference to the existing mdev api for our use is that the
devices don't report all generally available devices, only the
createable ones. So if a user wants to configure a VM, the selection is
restricted by what may currently run on the GPU (depending ont the exact
settings, e.g. mixed mode gpus where different models can be mixed on a
single GPU; not the default though)

We could overcome this, when we'd parse the general info from the
'nvidia-smi' tool, though I'm currently unsure if that interface is
stable and intended to be parsed (there is no json output or similar
AFAIK)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>
2024-10-24 18:33:46 +02:00
Dominik Csapak
eff59571be sysfstools: file_write: properly catch errors
since `print` is doing buffered IO, we don't always get an error there,
even if the underlying write does not work.

To properly catch that, do an unbuffered `syswrite` which circumvents
all buffers and writes directly to the file handle.

We aren't actually interested in the specific error here, but only if
the write was successful or not.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
2024-10-24 18:31:04 +02:00
Thomas Lamprecht
5b0106ebee bump version to 8.2.5
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-19 16:47:23 +02:00
Thomas Lamprecht
c4945bf96b tools: load PerlIO explicitly to avoid odd failures
Since commit ef0bcc9 ("tools: file_set_contents: use syswrite instead
of print") we're using PerlIO's scalar layer to ensure we encode any
potential unicode before passing the data to syswrite, which does not
support writing code points above 255.

Add an explicit use statement for PerlIO::scalar to avoid some odd
failures.

Some more background on why this seems odd:

After the pve-common version that included this change got moved to
public repos we got some reports in our Forum about CTs failing to
start [0] due to the lxc-pve-prestart-hook failing with an error
message like:

> Can't locate PerlIO.pm in @INC [...] at /usr/share/perl5/PVE/Tools.pm line 293.

Which points to the recently added `open(my $data_fh, '>', \$data)`
line in file_set_contents. The call chain from there upwards was
$lxc_setup->ct_file_set_contents <- $lxc_setup->pre_start_hook <-
closure <- lxc_hook <- lxc-pve-prestart-hook.
This seemed especially odd as we use `file_set_contents` in a lot of
other places and there was no all to obvious breakage from the change
on our test systems.

During evaluation I noticed some additional strange behavior, if one
called `file_set_contents` inside the closure before the call to
`pre_start_hook`, the error just goes away and one can observer that
%INC, which contains all loaded modules, suddenly does have a entry
for the PerlIO module, or well, it's scalar layer, which it did not
have without that call. So why the PerlIO can get automatically loaded
just fine most of the time but not inside the `pre_start_hook` is not
really clear yet; still loading PerlIO explicitly makes the issue go
away and seems sensible, so go for that and keep a comment to remind
more explicitly of this oddity. Once it's explained it can be removed
with a commit that mentions the explanation.

Further, if the PerlIO scalar layer cannot be loaded, the result is
that the passed reference is used as filename, which is far from
ideal, see the report in the perl GH [1] and the PR that fixes this
[2] by moving PerlIO::scalar into perl core proper, which will be
available in Perl v5.40 and thus our next Debian Trixie based major
release. This might have well to do with the original symptom that
embarked me on this odd (and not 100% finished) quest..

[0]: https://forum.proxmox.com/threads/156188/
[1]: https://github.com/Perl/perl5/issues/21275
[2]: https://github.com/Perl/perl5/pull/21282

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-19 16:47:23 +02:00
Thomas Lamprecht
e4394777d2 tools: code-style cleanup file_set_contents
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-19 16:42:43 +02:00
Thomas Lamprecht
f291dcd493 bump version to 8.2.4
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-16 19:02:23 +02:00
Thomas Lamprecht
f1fe7a0733 file set contents: fix error handling with or-operator precedence
In perl the `or` and the `||` operator do mostly the same thing but
with a different precedence level [0].

A statement like:
`$foo += bar() or die "error"`
is basically equivalent to:
`($foo += bar()) or die "error"`

That means as long as bar only returns zero or positive integers the
`or die` can only happen the first time, as otherwise $foo is bigger
than zero and thus will never evaluate to false. This can be
reproduced by perl -we 'my $foo = 1; $foo += 0 or die "wont happen";'

While one could switch to the `||` operator, this is a bit to subtle,
so to fix this, separate tracking the total bytes written from getting
the bytes written by the current call, this avoids the error potential
completely.

[0]: https://perldoc.perl.org/perlop#Logical-or-and-Exclusive-Or

Reported-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-10-14 11:14:23 +02:00
Filip Schauer
ef0bcc98ad tools: file_set_contents: use syswrite instead of print
The use of `print` can be inefficient for writing larger files due to
its default buffering in 8 KiB blocks.

This is especially problematic on `pmxcfs` where files are written in
4 KiB blocks due to the defaults of `libfuse2`. This leads to
significant write amplification on files larger than 4 KiB.

Patch (fix #5728: pmxcfs: allow bigger writes than 4k for fuse) [1]
addresses this by enabling `big_writes`, allowing up to 128 KiB blocks.
But due to the use of `print` in `file_set_contents`, writes are still
only buffered in 8 KiB blocks.

To further address this, this commit switches to using `syswrite`
instead of `print` to mitigate the block size limit imposed by `print`.
Combined with patch [1], file writes to `/etc/pve/` are now buffered in
128 KiB blocks.

The table below illustrates the drastic reduction in write
amplification when writing files of different sizes to `/etc/pve/` using
`file_set_contents`:

           print                big_writes+print     big_writes+syswrite
file size  written     amplif.  written     amplif.  written    amplif.
    1 KiB      48 KiB     48.0      45 KiB     45.0     41 KiB     41.0
    2 KiB      48 KiB     24.0      45 KiB     22.5     62 KiB     31.0
    4 KiB      82 KiB     20.5      80 KiB     20.0     73 KiB     18.3
    8 KiB     121 KiB     15.1      90 KiB     11.3     89 KiB     11.1
   16 KiB     217 KiB     13.6     146 KiB      9.1    113 KiB      7.1
   32 KiB     506 KiB     15.8     314 KiB      9.8    158 KiB      4.9
   64 KiB    1472 KiB     23.0     826 KiB     12.9    259 KiB      4.0
  128 KiB    5585 KiB     43.6    3765 KiB     29.4    452 KiB      3.5
  256 KiB   20424 KiB     79.8   10743 KiB     42.0   2351 KiB      9.2
  512 KiB   86715 KiB    169.4   43650 KiB     85.3   3204 KiB      6.3
 1024 KiB  369568 KiB    360.9  187496 KiB    183.1  15845 KiB     15.5

Since `file_set_contents` also performs a `rename` after writing, the
following table shows the results when the file is written without
renaming it afterwards:

           print                big_writes+print     big_writes+syswrite
file size  written     amplif.  written     amplif.  written     amplif.
    1 KiB      29 KiB     29.0      29 KiB     29.0     25 KiB      25.0
    2 KiB      29 KiB     14.5      30 KiB     15.0     25 KiB      12.5
    4 KiB      37 KiB      9.3      44 KiB     11.0     41 KiB      10.3
    8 KiB      61 KiB      7.6      45 KiB      5.6     45 KiB       5.6
   16 KiB     143 KiB      8.9      86 KiB      5.4     57 KiB       3.6
   32 KiB     396 KiB     12.4     225 KiB      7.0     69 KiB       2.2
   64 KiB    1281 KiB     20.0     673 KiB     10.5    105 KiB       1.6
  128 KiB    4789 KiB     37.4    3478 KiB     27.2    169 KiB       1.3
  256 KiB   18868 KiB     73.7    9976 KiB     39.0    572 KiB       2.2
  512 KiB   79304 KiB    154.9   42714 KiB     83.4   2150 KiB       4.2
 1024 KiB  347929 KiB    339.8  182483 KiB    178.2  11133 KiB      10.9

[1] https://lists.proxmox.com/pipermail/pve-devel/2024-September/065396.html

Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
2024-10-14 10:23:50 +02:00
Thomas Lamprecht
aeff857f77 bump version to 8.2.3
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-09-23 10:24:00 +02:00
Fabian Grünbichler
e83d895594 method schema: rename 'download' parameter to 'download_allowed'
to make it more obvious what it does, but keep old variant for now.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-09-23 10:19:20 +02:00
Fabian Grünbichler
c1ed9e3ee6 download handling: adapt cycle check to check value
instead of whether the key exists in the schema instance, just in case somebody
wants to set "download => 0".

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-09-23 10:19:20 +02:00
Fabian Grünbichler
6dbb3e0e78 schema: adapt description of method's download key
with the newly introduced strict checks in the API handler, the download key
actually marks which endpoints/methods are allowed to use the download
functionality of the REST server, and the "directly return filename to be
downloaded" part is no longer supported as a result.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-09-23 10:19:20 +02:00
Thomas Lamprecht
e068c3d30a cli handler: print correct command prefix for alias in asciidoc output
Otherwise only the base exename is printed, even if the alias is for
sub-command.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-07-31 20:24:04 +02:00
Thomas Lamprecht
7f3dc635d8 bump version to 8.2.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-07-30 21:42:16 +02:00
Fabian Grünbichler
48da50c2be Revert "section config: document package and its methods with POD"
This reverts commit d41bd420f8.
2024-07-24 13:44:45 +02:00
Fabian Grünbichler
b3ed595212 Revert "section config: update code style"
This reverts commit 29292d2a5d.
2024-07-24 13:44:40 +02:00
Fabian Grünbichler
50f0b6d0be Revert "section config: clean up parser logic"
This reverts commit 6cba8d7660.
2024-07-24 13:44:37 +02:00
Thomas Lamprecht
4386716c5a REST handler: use double-hyphen as argument prefix for doc and CLI help output
Use "modern" double-hyphen style, we're not some odd BSD derived
tool..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-07-22 19:10:28 +02:00
Wolfgang Bumiller
b64a40a506 fix #5529: cgroup: correctly handle change_cpu_quota without a quota
The function can be called with
- neither quota nor period
- only a period (quota will be 'max')
- both

$quota was therefore defaulted to 'max' and the check for whether
values were provided should use $period instead of $quota.
Also move the defaulting-assignment into the condition for clarity.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-07-22 19:07:36 +02:00
Dominik Csapak
06f436f126 fix #5486: tools: encode_text: add '%' to list of encoded characters
all text that is going through encode_text will at a later point be
decoded by 'decode_text'. The latter is decoding all percent encoded
characters, even those not originally encoded by 'encode_text'.

This means, to preserve the original data, we first have to at least
percent encode the '%' itself, otherwise it's impossible to properly
store e.g. '%20' there.

It would get saved as '%20' directly, but on the next read, it gets
decoded to ' ', which is not the original data. instead we have to save
it as '%2520', which gets then correctly decoded to '%20' again

This is especially important for the vm/ct/node description, as there
users can store external links, which already include percent encoded
characters.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2024-07-04 10:54:48 +02:00
Max Carrara
6cba8d7660 section config: clean up parser logic
In order to make the parser somewhat more maintainable in the future,
this commit cleans up its logic and makes its control flow easier to
follow.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-07-04 10:54:48 +02:00
Max Carrara
29292d2a5d section config: update code style
Replace `foreach` with `for` and use postfix deref instead of block
(circumfix) dereference (`$foo->%*` instead of `%$foo`).

Furthermore, make `format_config_line` a private sub instead of
unnecessarily declaring it as an anonymous subroutine, which avoids
the `&$sub_ref(...)` syntax altogether.

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-07-04 10:54:48 +02:00
Max Carrara
d41bd420f8 section config: document package and its methods with POD
Apart from the obvious benefits that documentation has, this also
allows LSPs to provide docstrings e.g. via 'textDocument/hover' [0].

Tested with Perl Navigator [1].

[0]: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_hover
[1]: https://github.com/bscan/PerlNavigator

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-07-04 10:54:48 +02:00
Jing Luo via pve-devel
6dc7a73bd5 tools: fix syscall mknod()
b792e8df81 introduced a bug that can cause this:

Undefined subroutine &PVE::Syscall::SYS_mknod called at /usr/share/perl5/PVE/Syscall.pm line 11

It should be mknod, not SYS_mknod. This caused other pve perl lib failing
to build. I couldn't reproduce this on amd64 build, but I could reproduce this
on arm64 build; however this didn't seem to fix the issue, unless I revert
b792e8df81.

cf: b792e8df81
Signed-off-by: Jing Luo <jing@jing.rocks>
2024-07-03 10:25:21 +02:00
Thomas Lamprecht
1a6005ad23 bump version to 8.2.1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-23 15:43:01 +02:00
Wolfgang Bumiller
b518bbd5f9 interfaces: support stanzas without types/methods
This is allowed in ifupdown2 and previously interfaces named
'vmbr\d+' were recognized as bridges even if they used this mode.
With commit e68ebda4f1 this is no longer the case.

Fixes: e68ebda4f1 ("fix #545: interfaces: allow arbitrary bridge names in network config")
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-04-23 15:19:43 +02:00
Thomas Lamprecht
a061517cdd bump version to 8.2.0
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-21 11:51:17 +02:00
Stefan Hanreich
e68ebda4f1 fix #545: interfaces: allow arbitrary bridge names in network config
Similar to other interface types, we can detect a bridge by the
presence of its bridge_ports attribute, rather than solely relying on
the "vmbr" ifname prefix heuristic. For OVS bridges we need to examine
the OVS type instead.

The check needs to be moved up since other prefixes could
theoretically be included in a bridge name and then would otherwise
get picked up wrongly.

Also added a warning for interfaces named vmbrX that are not bridges
to catch possible misconfigurations.

Originally-by: Jillian Morgan <jillian.morgan@primordial.ca>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
2024-04-21 11:22:10 +02:00
Thomas Lamprecht
55ba8536f0 bump version to 8.1.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2024-04-17 21:10:39 +02:00
Fiona Ebner
c302a28a21 json schema: add format description for pve-storage-id standard option
so that the option can be used as part of a property string.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
2024-04-11 17:19:52 +02:00
Folke Gleumes
a656f674c2 docs: add missing prefix
include 'PVEAPIToken=' prefix in the example for target-endpoint which
is mainly used for remote migrations.

Signed-off-by: Folke Gleumes <f.gleumes@proxmox.com>
2024-04-08 18:01:35 +02:00
Wolfgang Bumiller
10349969cc bump version to 8.1.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-06 12:03:12 +01:00
Max Carrara
ff39327769 ticket: remove fallback for SHA1-base64 CSRF prevention tokens
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
2024-03-06 12:03:10 +01:00
Wolfgang Bumiller
68b234f35e add PVE::Systemd::is_unit_active
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-03-06 11:57:22 +01:00
Wolfgang Bumiller
c6ec71d846 schema: fixup description vs format_description in remote_format
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2024-01-22 12:52:41 +01:00
Fabian Grünbichler
f869c96a94 network tests: test #5141
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-01-03 12:18:26 +01:00
Fabian Grünbichler
80ed66dc17 fix #5141: network parser: fix accidental RE result re-use
$1 and friends are not cleared if a RE fails to match, in which case they will
contain the captured values from a previous successful match in the same scope.

deduplicate the two branches to avoid accidental re-introduction.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-01-03 12:16:14 +01:00
Fabian Grünbichler
fcc97ec96d network parser: iterate deterministically
makes the behaviour easier to analyze, and also helps when testing since it
allows constructing test cases that trigger certain order of parsing.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-01-03 12:15:37 +01:00
Fabian Grünbichler
e8c1bbfca2 network tests: switch to ifupdown2
adapt allow-* to auto, and drop the one test where behaviour is not testable
anymore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2024-01-03 12:14:43 +01:00
Wolfgang Bumiller
63b74c5089 expose SYS_prctl
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2023-12-14 10:09:48 +01:00