as is already supported by the UI (and PBS).
A nice bonus is that warn() can be used by both workers and non-workers. For
workers, the output is redirected/duplicated as set up by {fork,tee}_worker(),
and non-erroring workers that issued a warning will end in a WARNINGS state.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Shellquote is needed for '~', and while it doesn't help with '-',
there should be no problem, because options are separated from mailto
since commit 216a3f4f13.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
sync_mountpoint takes a path, gets an open filedescriptor and calls
syncfs(2) on it.
by opening with O_PATH the syncfs call fails with EBADF (see open(2)).
found by running:
```
pkill -f 'pvedaemon worker';
strace -yyttT -s 512 -o /tmp/trace -fp $(pgrep -f pvedaemon$)
```
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
This patch enables the sendmail helper sub to send emails with a non-existant
display name in the from address. This is used to replace the direct call to
the sendmail binary in pve-manager/PVE/API2/APT.pm.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
PVE::Tools::sendmail currently always sends a multipart/alternative message
irrespective of the actual content of the mail (e.g. a plain-text only mail
need not be sent as multipart message).
Additionally a few small refactorings based on the discussion
in https://lists.proxmox.com/pipermail/pbs-devel/2020-August/000423.html
and commited in 66004f22c6475ceb0146cf2df1f380f9f0274be4 in the
rust proxmox repository git://git.proxmox.com/git/proxmox.git
were carried over.
tested by creating a backup of a VM and setting an e-mail address, having
ha-manager send a mail after fencing as well as sending a few small mails via
'perl -e'
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
to call out/err/logfunc with each line, we search for a newline and call
outfunc/logfunc with everything before that
since we do a select/read (with 4096 size) in a loop, this means
that if we have very long lines, we search for a newline in an
ever growing buffer (for which we know does not contain a newline)
so instead, only search the new data for newlines
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this is (via file_get_contents and file_copy) used for reading from
pmxcfs, which has a file size limit of 512k. since quite a number of
call sites would need to explicitly override this (and then get updated
if we bump the limit on the pmxcfs side again in the future), making our
default file reader compatible by default seems the better solution.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
by introducing a safe_compare helper. Fixes warnings, e.g.
pvesh get /nodes/<NODE>/network
would print "use of uninitialized"-warnings if there are inactive
network interfaces, because for those, 'active' is undef.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Tested-by: Oguz Bektas <o.bektas@proxmox.com>
if the size/avail of a mount is bigger than a certain amount,
json_encode writes the number in scientific format, which was not
matched by our \d+ regex.
This then resulted in 'undef' values for the result hash and
subsequently led to errors and warnings.
Extend the regex to also match scientific formatted numbers,
perl can then use them as is, no need for any conversion.
https://forum.proxmox.com/threads/bug-in-pve-tools-df-when-adding-petabyte-scale-storage.60090/#post-277050
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The current version had only one user in LXC, so move the LXC-specific
code there to reuse this in QemuServer.
Also cache, since the host's architecture can't change during runtime.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
and constant AT_EMPTY_PATH for chowning a directory/file opened via
openat(2), for example when walking/creating a directory tree without
following symlinks.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
See `man 2 setresuid`. The code was tested with small UIDs (109) and
one which does not fit into 16 bit (100000000), since I wasn't too
sure about the workings of setresuid vs. setresuid32 (see NOTES of
the manpage) - it worked with both.
Linux on amd64 has only a single setresuid, as it the 64 bit arch
came after the setresuid32 syscall, and thus it started with the
later one as single common one
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Extract major/minor from `dev_t` values as found in stat()
calls, with support for the full 32 bit values.
The device value returned by stat() is 32 bits long and encoded as
high 12 bit of the minor value as in the 12 MSBs, 12 bit major value,
then the low 8 bit of the minor value in the low byte.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
giving split_list the string "0" would actually return
an empty list instead of a list with the element "0"
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
To avoid warnings (and in the future, errors) like:
> Unescaped left brace in regex is deprecated here (and will be fatal
> in Perl 5.32), passed through in regex; marked by <-- HERE in
> m/([^{]+)?({ <-- HERE ([^}]+)})?/ at /usr/share/perl5/PVE/Tools.pm
> line 673.
with future perl versions, like Debian Buster for example has.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
when stopping a worker while doing a run_fork_with_timeout,
we want to handle that there and send the child the SIGTERM
so that it can clean up
for this we have to use readline_nointr,
because the read from the pipe gets interrupted by the signal
this partially fixes#1874
as we now correctly clean up the new disk, except if it is
on lvm/lvmthin (possibly other storages as well), and use the old disk
in the config
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this is a wrapper to have an uninterruptible readline
so that we can read e.g. from a pipe even if interrupted by
a signal
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
We cannot properly encode blessed objects as json, so
instead, we should stringify them. This happened for
instance if a VM's systemd scope wasn't cleaned up as we
got an error as a Net::DBus::Error object causing a
"malformed json string" error to appear instead of the
actual message.
Additionally, add a 'must_stringify' helper: The above error
object implements a '""' operator for stringification (as
all error should), but in theory that could die as well, in
which case we just return a generic error string we'll
hopefully never see...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
since we sometimes use their length in a format string for printf
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Without this patch we printed to STDOUT and STDERR, respectively, if
no $outfunc or $errfunc was passed.
Sometimes it's useful if one, or even both, of those prints can
be suppressed, currently this can only be done by either using an array
of arrays or a whole string for the command and redirecting STDOUT
and STDERR.
Add a 'quiet' option which allows to do this in an easier way.
It allows to silent STDERR or STDOUT or both.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
converting from 0.5 gb to mb resulted in 0 mb
with this patch it correctly returns 512
also add tests and catch more errors
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Most times a port was requested for a specified IP family (v4, v6)
only. Thus also ensure that the port from the respective family got
ready, else we may return on a false positive.
As we had no user setting the $timeout param we can add the $family
param as second one, it'll get used more often, so no need to put it
at the back.
As we do nothing if not defined this does not changes the behavior of
our users yet.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
When we do not instantly get the lock we print a respective message
to stderr. This shows also up in the task logs, and if it's the last
message before a 'Task OK' the UI gets confused an shows the task as
erroneous.
Keep the message as its a good feedback for the user to see why an op
seems to do nothing, so simply add a trailing newline.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Encode the result or the error in JSON. This way complex objects or
exceptions may be passed to the parent in a generic way.
This allows to remove the second pipe 'pipe_err'.
Allow also to return undef without any warnings to our caller.
This avoids a "use of uninitialized variable ..." warning
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We often need to convert between file sizes, for formatting output,
but also code-internal. Some methods expect kilobytes, some gigabytes
and sometimes we need bytes.
While conversion from smaller to bigger units can be simply done with
a left-shift, the opposite conversion may need more attention -
depending on the used context.
If we allocate disks this is quite critical. For example, if we need
to allocate a disk with size 1023 bytes using the
PVE::Storage::vdisk_alloc method (which expects kilobytes) a
right shift by 10 (<=> division by 1024) would result in "0", which
obviously fails.
Thus we round up the converted value if a remainder was lost on the
transformation in this new method. This behaviour is opt-out, to be
on the safe side.
The method can be used in a clear way, as it gives information about
the source and target unit size, unlike "$var *= 1024", which doesn't
gives direct information at all, if not commented or derived
somewhere from its context.
For example:
> my $size = convert_unit($value, 'gb' => 'kb');
is more clear than:
> my $size = $value*1024*1024;
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
perls 'local' must be either used in front of each $SIG{...}
assignments or they must be put in a list, else it affects only the
first variable and the rest are *not* in local context.
This may cause weird behaviour where daemons seemingly do not get
terminating signals delivered correctly and thus may not shutdown
gracefully anymore.
As we only send SIGINT to processes if a manual stop action gets
triggered just catch this one here.
As this is a general method which allows to pass an arbitrary code
payload we cannot sanely handle all signals here, so remove trapping
all other besides SIGINT, if those need to be trapped that should be
done by the caller on a case by case basis.
Fixes: #1495
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We run into problems where this method returned to early, even if the
port wasn't actually ready yet. The reason for this is that we
checked /proc/net/tcp which does not guarantees and always up to date
state of only those ports which are actuall available, i.e. a port
could linger around (time-wait state) or appear even if it wasn't
accepting connections yet (as stated in the kernel docs:
/proc/net/tcp is seen as obsolete by the kernel devs).
Use the `ss` tool from the iproute2 package, it uses netlink to get
the current state and has switches where we can direct it to really
only get the state of those sockets which interest us currently.
I.e., we tell it to get only listening TCP sockets from the requested
port.
The only drawback is that we loop on a run_command, which is slower
than just reading a file. A single loop needs about 1ms here vs the
60µs on the /proc/net/tcp read. But this isn't a api call which is
used highly frequently but rather once per noVNC console open.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Otherwise perl tries to bind+listen on a UDP socket if the
TCP socket fails - which is a waste since we're looking for
TCP ports.
Additionall since UDP doesn't support listen(), perl will
return EOPNOTSUPP instead of, say, EADDRINUSE. (We don't
care about the error in this code though.)
While it should be impossible to bind to a wildcard address
when the port is in use by any other address there's one
case where this is allowed, and that's when the port is in
use by an ipv6 address while trying to bind to an ipv4
wildcard.
This currently happens when qemu finds ::1 for the
'localhost' we pass to qemu's spice address while we're
resolving the local nodename via IPv4.
Raw syscall numbers were not platform independent, so replace them
with the helpers provided from the syscall.ph perl bits helper.
This makes reading the code easier as a nice side effect.
As syscall.ph is not an ordinary module and makes problems when it is
required by multiple modules we make a own module PVE::Syscall which
loads it and allows to export the necessary constants in a sane way.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Previously an external exception (eg. caused by a SIGARLM in a code
which is already inside a run_with_timeout() call) could happen in
various places where we did not properly this situation.
For instance after calling $lock_func() but before reaching the cleanup
code. In this case a lock was leaked.
Additionally the code was broken in that it used perl's automatic hash
creation side effect ($a->{x}->{y} implicitly initializing $a->{x} with
an empty hash when it did not exist). The effect was that if our own
time out was triggered after the initial check for an existing file
handle inside $lock_func() happened (extremely rare since perl would have
to be running insanely slow), the cleanup did:
if (my $fh = $lock_handles->{$$}->{$filename}->{fh}) {
This recreated $lock_handles->{$$}->{$filename} as an empty hash.
A subsequent call to lock_file_full() will think a file descriptor
already exists because the check simply used:
if (!$lock_handles->{$$}->{$filename}) {
While this could have been a one-line fix for this one particular case,
we'd still not be taking external timeouts into account causing the
first issue described above.
When the child process running the command got an signal or failed
to execute exitcode was still undefined as we extract it just only
after the signal/failed to execute check.
This led to:
> Use of uninitialized value in numeric ne (!=) at
> /usr/share/perl5/PVE/API2/Qemu.pm line 1433.
errors if we used run_commands `noerr` param and checked for the
commands exit code.
So just default the exit code to -1 for such cases.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
since the "lang" param has not worked, introduce a "keeplocale"
parameter instead.
the default behaviour is the same (set LC_ALL to 'C'), but we can use
the parameter to keep the locale from the host (eg. for the vncshell)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when reserving ports, we use lock_file to lock the
reservation file, but then use file_set_content which
writes a new file and renames it, making the lock invalid
and different processes waiting for the lock get inconsistent
data
instead we use a designated lock file for the lock, so that we don't
lose the lock when writing the reservation file
this should fix the problem that sometimes multiple vms get the
same vnc/spice port
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Commit de9a267 introduced vec() to optimize the generation
by using binary operations instead of converting back and
forth between hex and strings, but forgot to switch over to
the binary sha1 method. This resulted in only the first 6
hex digits of the output string making up the address.
This essentially performas the task of systemd-run while
also waiting for the job to finish.
With the systemd-run version in jessie we run into a race
condition where the executed process can start forking child
processes before the systemd daemon is done setting up the
scope's cgroups, causing the children to NOT be included in
the cgroups. This means the child processes (in our case
qemu) will not adhere to the limits we want to apply to it
via cgroups.
enter_systemd_scope() performs the setup task of systemd-run
and waits for the job to finish, after this we can spawn the
qemu process without systemd-run.
This was already implemented in PVE::LXC::lock_aquire() and
lock_release(). Enabling refcounting in the general
PVE::Tools::lock_file() and lock_file_full() methods allows
us to use one code base for flocking.
Furthermore, we could get rid of various xx_no_lock methods
that were required because the old non-refcounting version
did not support nested flocks (the inner most flock would
close the file handle and thus release the flock).
If we can't acquire the lock in lock_file_full and get interrupted
by a signal inqeual to EINTR (e.g. SIGTERM), output also it's name
in the error message to allow better debugging.
Also fix a typo.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Allow to return the exit code of the executed command.
And as we do not reach the return of the exit code if it was not 0,
a noerr parameter is also needed so we can suppress the 'command
failed' die in case of an exit code unequal to 0.
This is required as some programs return another value than 0 when
they succeed, For example `systemctl list-jobs` returns a value
>= 0 on a successful execution, normally 1.
Without this patch a run_command call to `systemctl list-jobs` gets
marked as failed although it was successful.
This does not break current behaviour in any way as setting the
noerr parameter is required to return something other than 0 or
undef, which are equal in a boolean comparison.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Most syscall wrappers in perl return 1 on success and our
current use of Tools::unshare isn't using the return value
(yet), so let's fix this while we can.
Also it seems to make sense to use prototyping on syscalls
to add some compile-time argument checking.
In an alternation /a|b|c/ the first match matches, so while
'1.1.1.121' matches /^$IPV4RE$/ (note the ^ and $ anchors),
parsing a line like /nameserver ($IPV4RE)/ would only
extract '1.1.1.12', ignoring the last '1' due to the /[1-9]/
alternative matching before the /1[0-9]/ one.
