Else, a user would need to renew it first before being able to revoke
it, which does not make much sense..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this normally just means that the old cert is already expired, we do
not care for that - after all: we got a new (renewed) valid cert
successfully.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
If source is missing, pvesr will set it via job_status
on the next run. But the info is already present here,
so it doesn't hurt to use it.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
reload is actually preferred, and even if most of the time this even
won't reach the API, allowing to start them is still definitively
fine!
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
In some situations Ceph's auto-detection doesn't recognize the device
class correctly. The option allows to set it directly on osd create,
instead of altering it afterwards. This way the cluster doesn't need to
shift data back and forth unnecessarily.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Adds a new api endpoint at cluster/backupinfo for cluster wide backup
stuff. This is necessary because cluster/backup expects a backup job ID
at the next level and thus other endpoints are hard to impossible to
implement under that hierarchy.
The only api endpoint available for now is the `not_backed_up` which
returns a list of all guests which are not covered by any backup job.
The top level index endpoint is left unsused for now to be available for
a more generic summary endpoint in the future.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
This patch adds a new API endpoint that returns a list of included
guests, their volumes and whether they are included in a backup.
The output is formatted to be used with the extJS tree panel.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
The `guest include` logic handling `all` and `exclude` parameters was in
the `PVE::VZDump->exec_backup()` method. Moving this logic into the
`get_included_guests` method allows us to simplify and generalize it.
This helps to make the overall logic easier to test and develop other
features around vzdump backup jobs.
The method now returns a hash with node names as keys mapped to arrays
of VMIDs on these nodes that are included in the vzdump job.
The VZDump API call to create a new backup is adapted to use the new
method to create the list of local VMIDs and the skiplist.
Permission checks are kept where they are to be able to handle missing
permissions according to the current context. The old behavior to die
on a backup job when the user is missing the permission to a guest and
the job is not an 'all' or 'exclude' job is kept.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
otherwise the ACME endpoint might return the ordered domain in lower
case and we fail to find our plugin config.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
allow users with Sys.Modify to modify custom or ACME certificates. those
users can already hose the system in plenty of ways, no reason to
restrict this in particular to being root@pam only.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
As a first step to make the whole guest include logic more testable the
part from the API endpoint has been moved to its own method with as
little changes as possible.
Everything concerning `all` and `exclude` logic is still in the
PVE::VZDump->exec_backup() method.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
to add the pg_autoscale_mode since its activated in Ceph Octopus by
default and emmits a waring (ceph status) if a pool has too many PGs.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Luminous, Nautilus and Octopus. In Octopus the mon_status was dropped.
Also the ceph status was cleaned up and doesn't provide the mgrmap and
monmap.
The rados queries used in the ceph status API endpoints (cluster / node)
were factored out and merged to one place.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
and do a breaks on older network package as we do not depend on it
due to it being an optional/experimental feature, so reverse the
depends with the breaks.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
since this API endpoint is used for the node selector in the GUI, which
causes quite widespread breakage.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
we need to parse the config even if it does not exist - it will return
the 'standalone' entry that's needed to be backwards compatible with
existing setups.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
which returns a list of challenge api types with the schema of their
required data (if it exists)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ Thomas: adapt to my changes from proxmox-acme schema def and change
path from challengeschema to challenge-schema ]
As proxmox-acme has now a default delay for DNS challenge plugins,
which is the important one. Those are just for not overloading the
acme servers with a lot of requests, but once the challenge was
propagate they have it verified pretty quickly, so reduce delay for
checking validation after first requesting it down to 10 seconds
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
re-loading it always would mean that we could potentially switch the
config to something completely different, and the mix of the previous
and the old could result in total bogus actions.
Better to use the same one for one full order, even if it may get
"outdated" it was still valid in the past and most important
coherent.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Drop various leftovers from the storage content API module this was
based on, e.g., ACME plugins have no fixed options and the like.
Also, the descriptions shouldn't mention "storage".
Further, drop the "update_config" "helper" with its operations
effectively only increasing code complexity and adding another rabbit
hole to jump into.
IF, this should have been factoring out the lock+read+write cycle
only, living the rest to a passed CODE-ref, but honestly that saves
only really the read and write config lines, and at this point
nothing is really gained, so just let it be.
Should have been actually three or so separate patches, but to deep
into this rabbit hole to care..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
for now mostly due to the "nice" property of the acmedomains which
do not use their property key as index but actually the doamain.
Without this one could set up duplicated domain entries just fine,
but once using them -> error.
This is not nice UX, so verify node config before writing an updated
one out, to catch those issues.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
add checks, encoding of loaded data files, update API path, proper inclusion into API tree
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
instead of relying that the authorization URLs and the ordered
identifiers are sorted the same way for already validated
authorizations.
on the contrary, RFC 8555 even says:
"The authorizations required are dictated by server policy; there may
not be a 1:1 relationship between the order identifiers and the
authorizations required."
authorizations MUST always include a single identifier, no matter which
state they are in.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
mixed bag of improvements:
- return hash based on domain
- check for duplicate domains, allow overlap between main and additional
keys otherwise
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
With this configuration it is possible to use many different plugins
with different providers and users.
Signed-off-by: Wolfgang Link <w.link@proxmox.com>
The public_addr option for creating a new MON is only valid for manual
startup (since Ceph Jewel) and is just ignored by ceph-mon during setup.
As the MON is started after the creation through systemd without an IP
specified. It is trying to auto-select an IP.
Before this patch the public_addr was only explicitly written to the
ceph.conf if no public_network was set. The mon_address is only needed
in the config on the first start of the MON.
The ceph-mon itself tries to select an IP on the following conditions.
- no public_network or public_addr is in the ceph.conf
* startup fails
- public_network is in the ceph.conf
* with a single network, take the first available IP
* on multiple networks, walk through the list orderly and start on
the first network where an IP is found
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
so that we can choose to only include the versions and not all metadata
this is done to avoid having a seperate 'versions' api call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of having multiple regexes in various places for the name,
define a 'SERVICE_REGEX' in PVE::Ceph::Services, and use that
everywhere in the api where we need it
additionally limit new sevices to 200 characters, since
systemd units have a limit of 256 characters[0] (including suffix), and
200 seems to be enough.
users can now create ceph services on machines with hostnames
longer than 32 characters
0: https://www.freedesktop.org/software/systemd/man/systemd.unit.html
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the network and the cluster network are optional in the ceph config
and with 'pveceph init', so only check if we have an ip address
from those networks if it is actually configured
otherwise, the createosd call dies with an 'ip' error message
even if it would work
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
To (re)start and stop Ceph services the API calls systemd units using the
function ceph_service_cmd. If unspecified, this function assumes the type
".target" for a unit. By making the unit type suffix in the API optional, it
can make use of this assumption.
Signed-off-by: Dominic Jäger <d.jaeger@proxmox.com>
We need a recent version *and* our patches on top of the vanilla
ifupdown2:
0001-don-t-remove-tap-veth-fwpr-interfaces-from-bridge-on.patch
0004-don-t-remove-bridge-is-tap-veth-are-still-plugged.patch
So check for at least 1.2.8 and the pve string in the version output.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It's possible to have a situation where the cluster network (used for
inter-OSD traffic) is not configured on a node. The OSD can still be
created but can't communicate.
This check will abort the creation if there is no IP within the subnet
of the cluster network present on the node. If there is no dedicated
cluster network the public network is used as a failsafe even though
this situation should not occur.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
Ceph MGR fails to start when installed on a node without existing
symlink to /etc/pve/ceph.conf.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
To make it backward compaitble. NBo real harm without this, but lots
of ugly undefiend $val warnings...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Just because OVS is installed it doesn't mean that OVS interface
(changes) are configured - so check for that.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the code returns undef in case there is no 'tos', and the code
calling this api call handles a non-existing tos already, but
fails in that case becasue of the failing return value verification
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
It's a bit hard to figure out the exact constellation required for
this to happen, but we saw it in live systems when one node was dead
in a three node cluster.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
include the version as string and as parts, as we do the split
already. Also include the build commit, so if we re-release a ceph
version, we can differ here too.
Use node as key, to make the new entry a bit more general, could be
easily expanded with other infos, if required.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Removed in commit fcb8022169 as we
wanted to re-use Debian Busters upstream version, but we re-uploaded
our own again. And besides that, this version would be still
interesting if it was not uploaded by us..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This moves the cfs register code for vzdump.cron to the
pve-guest-common package. Therefore, it relies on the corresponding
patches in pve-guest-common and pve-docs as build dependencies.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
commit 2d2ed7ab53 had a valid cause but
unnecessarily used the static PVE::AccessControl::check_permissions.
As the RPCEnvironment based check method has a "$noerr" parameter and
we already have a rpcenv instance readily available, we can use that
one just fine.
this is the last caller of PVE::AccessControl::check_permissions(),
which is the last caller of PVE::AccessControl::permission(). both can
thus be dropped altogether.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It was intended that for partitioned disks, we create one and use it.
Instead the code died always when the disk was used and not of type 'LVM'
We now check correctly the 2 cases:
* used for partitions and has gpt
* used and lvm
The remaining api call handles those two cases correctly
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
If you updated a job in "exclude" mode with some VMIDs specified to "pool" mode,
the backup job would retain the "exclude" section and thus not back up all VMs.
The GUI misrepresents this, showing that all VMs will be backed up or
straight up break and show "exclude" mode again, with the backend still
being on "pool" - to prevent this, we always delete a jobs "exclude" list
when it's switched to "pool".
Co-authored-by: Tim Marx <t.marx@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
This was previously gated to CLI only, but it causes a vzdump job
started with the newly introduced "Run Now" button to fail if it
includes VMIDs on other nodes.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
use PUT for setting or unsetting, as POST/DELETE (like the old node
specific API used) makes no sense. One does not creates or deletes
the flag, they are always here. One just updates their value
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add a new private helper which just does that for us, to avoid having
the same, or multiple implementations doing the same thing.
Allow to pass the $rados for sharing the same connection but also
just create it else for convenience
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this api call can set multiple flags at once, but does this in a
worker since this can take quiet some time
also we only set/unset flags that are not already set/unset (respectively)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we want a GET api call where we return all available flags with
description, but the GET 'flags' api call already returns a string
and not an array, so we cannot use that
instead, we add a new api call that returns that feature and
with e.g. pve 7 we can remove the 'flags' api call and
rename 'flags2' to 'flags'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this is used to generate the paramters for the set/unset api call
and will be used for the parameters of the upcoming 'set multiple flags' call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we want to improve the version hints in the osd tree gui and need
the version at the host nodes
we could (and want to) workaround it in the gui to have that
info for both versions of the api call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
commit 970f96fdbb did not account for
getting the correct size parameter from the api call, so we ignored
it always resulting in uses not be able to set an explicit db/wal
size
Originally-fixed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead of silently ignoring them. since we are in a task worker here
this is especially important - otherwise the task status/result is also
wrong!
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
nautilus puts non running monitors also in the monmap, so only show
as running when it has quorum
this is also not 100% correct, but the only 'correct' alternative is
to try and get/parse the systemd status of the units and broadcast it
to the pmxcfs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this fixes an issue where only one monitor is in mon_host, which is
offline, prevents a client connection
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if auth is 'none' there is no client keyring, so do not generate it and
do not write it into the config
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
ms_bind_ipv4 is default true and osds look for both
ipv6 and ipv4 addresses in cluster network/public network
since we only allow for one network each (which must be either
ipv4 or ipv6) we disallow ipv4 if ipv6 is detected
this fixes not starting osds on an ipv6 enabled, newly-setup cluster
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when there is no 'public_network' in the config, the monitor
can only find an ip if it is given explicitly, either via commandline
(not possible with systemd) or via the ceph.conf
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
a 'mon remove' does this already for us, so do not stop it
this lead to a race where we could stop the next to the last monitor
before it was removed from the cluster, leading to a state
where two monitor were needed for quorum, but only one did exist
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we need to remove an ip, ip:port or a ipvector from monhost
so use multiple regex search and replaces for this
this looks not really nice, but due to the strange format
of the line (e.g. ',' is a seperator inside and outside of a vector,
also ipv6 adresses may be surrounded with [] but so are vectors),
i found no better way
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
otherwise it is possible that multiple users create monitors at the same
time, resulting in a wrong ceph.conf and probably worse
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in nautilus, the default msgr protocol is v2, but it has to be
explicitely given to monmaptool, also we don't want to use the
monitor sections anymore so only update mon_host
ceph can cope with mixed mon_host and monitor sections, so this is
not a problem
also the ceph-create-keys part is not necessary anymore since
this is done by the monitor itself now
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by using our new 'get_services_info'
this already checks for nautilus+ style 'mon_host' key in the ceph.conf
for the ip address
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it makes no sense to have the mon key inside the client.admin.keyring
also the order and operations did not make much sense
also create the client admin keyring when creating the config
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if we already have a monitor, we can try to get the public_network via
the ceph configuration database
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in a case where we cannot connect to any monitor, we did not get
any info even if we have them via the pmxcfs
so get the RADOS object in an eval, and get the info we have from the
config/pmxcfs, and set the state to unknown if we cannot query via RADOS
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since we do not support creating filestore osds anymore, drop
the journal size from the config
and move the keyring from global to client
this makes it possible to omit the osd keyring path
(which was the default but got overwritten from the global section)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
ifupdown2 reload can't work with openswitch until we implement
ovs.
I don't think that too much users are mixing ovs && bridge anyway.
It's possible to use ifupdown2 with ovs for ifup/down with ifupdown script,
but config need to be changed, and I don't have tested too much.
(maybe add a conflict in ifupdown2 package with openvswitch package for now)
Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
This was for vxlan interfaces and fixed in ifupdown2 with my last patches.
simply reload network, and if we still have errors, we can use ifquery to check them later
Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
using the new get_guest_config_property helper from pve-cluster,
which allows us to get this info with relatively low overhead.
With a somewhat realistic setup of 303 guest configurations here my
API call timing changes from ~ 24 to 26 ms without this to 26 to 28
ms with this patch applied, which seems reasonable.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ceph luminous does not use the 'name' property in the metadata
everywhere, so fall back to 'id'
this makes the ceph dashboard usable while having still luminous
(relevant for upgrading)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The aim of this patch is to reorder/rework the code of the api call
so that it gets more readable
it adds comments of what/why something is done, removes
code duplication between db/wal checks/creation
There are two changes in behaviour:
* when a device is given more than once via the api,
the user gets a parameter exception for the db or wal
with the information that the explicit defined devices must be
different
* we check the usage for db/wal before the worker, so that the user
gets instant feedback if a device is already in use
(this is more for api users than for gui users, since we do those
checks there also)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since the size of an LV can only be a multiple of 512b, we round
down to the next kib
we then have to mulitply it by 1024 for the partition, since
append_partition expects bytes and not kib
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
similar to the MDS api, so that DELETE and POST calls can operate on
the same path. This does not changes the CLI pveceph interface
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
As in a situation where we /had/ a manager but destroyed it this
key's value is a empty string, and if we pass that to the WebUI we
get strange results form of a ghost MGR entry with ExtJS auto-ID
generation as name -> pretty confusing.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
no point in first building a list if we can just remove it directly
afterwards, it's eval-ed anyway and $osd_list did not get touched
in-between.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
with this, osd destruction is left to ceph-volume if the osd was created
with ceph-volume, else our old code remains mostly the same since
we want to be able to destroy upgraded osds
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this completely rewrites the ceph os creation api call using ceph-volume
since ceph-disk is not available anymore
breaking changes:
no filestore anymore, journal_dev -> db_dev
it is now possible to give a specific size for db/wal, default
is to read from ceph db/config and fallback is
10% of osd for block.db and 1% of osd for block.wal
the reason is that ceph-volume does not autocreate those itself
(like ceph-disk) but you have to create it yourself
if the db/wal device has an lvm on it with naming scheme 'ceph-UUID'
it uses that and creates a new lv
if we detect partitions, we create a new partition at the end
if the disk is not used at all, we create a pv/vg/lv for it
it is not possible to create osds on luminous with this api call anymore,
anyone needing this has to use ceph-disk directly
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
as already announced over two months ago[0], remove the unofficial
SheepDog plugin now completely. Besides that it was never fully
supported in Proxmox VE one of its main developer and ex-maintainer
declared it as abandoned[1], and thus just let's remove it, git
allows to resurrect it any time if a wonder happens anyway.
[0]: https://pve.proxmox.com/pipermail/pve-user/2019-March/170497.html
[1]: http://lists.wpkg.org/pipermail/sheepdog/2019-March/068449.html
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
since we will have a seperate gui for the manager, we do not need this
anymore
this is a breaking api change
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and use the broadcast when a service is added/removed
we will use 'get_cluster_service' in the future when we generate a list
of services of a specific type
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
add two new api calls in /cluster/ceph
status:
the same as /nodes/NODE/ceph/status, but accessible without
nodename, which we don't need, as in the hyperconverged case, all nodes
have the ceph.conf which contains the info on how to connect to the
monitors
metadata:
combines data from the cluster filesystem about the services,
as well as the 'ceph YYY metadata' info we get from ceph.
with this info we can convieniently display which services exists,
which are running and which versions they have
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Wit commit a74ba607d4 we switched over
to using the dpkg-dev provided helpers to set package version,
architecture and such in the buildsystem.
But unlike other repositories we used the version also for giving it
back over the API through the during build generated PVE::pvecfg
module, which wasn't fully updated to the new style.
This patch does that, and also cleans up semantics a bit, the
following two changed:
release is now the Debian release, instead of the "package release"
(i.e., the -X part of a full package version).
version is now simply the full (pve-manager) version, e.g., 6.0-1 or
the currently for testing used 6.0-0+1
This allows to do everything we used this information for even in a
slightly easier way (no string concat needed anymore), and fits also
with the terminology we often used in our public channels (mailing
lists, forum, website)
Remove some cruft as we touch things.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
From Nautilus release changelog[0]:
> The auid property for cephx users and RADOS pools has been removed.
> This was an undocumented and partially implemented capability that
> allowed cephx users to map capabilities to RADOS pools that they
> “owned”. Because there are no users we have removed this support.
[0]: https://ceph.com/releases/v14-2-0-nautilus-released/
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This partially reverts commit f9b08743a5
as we had some wrong assumptions about lastentries and the other
params, so just note conflicts in the description but let the tool
itself make the checks
This reverts commit f9b08743a5.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this uses the new journalreader instead of journalctl, which is a bit
faster and can read from/to cursor and returns a start/end cursor
also you can give an unix epoch as time parameters
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
ceph nautilus changed the structure of 'pg dump osds'
they moved the data one level below
parse both new and old format, and bail if it returns anything else
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
otherwise this potentially returns outdated information (like the
cluster being quorate when corosync has crashed on all nodes 5 minutes
ago).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
It makes sense to not give users without Sys.Audit permissions to
much information over a node and this is relatively easy and cheap to
check and enforce at those two points.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Oguz Bektas <o.bektas@proxmox.com>
Reword the error message in find_mon_ip to make it more clear, that
there is no active IP configuration for the ceph public network.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
If calls aren't proxied to the selected node, which seems legit in
some cases, this will cause some misleading errors while ceph is
not installed on that node. Therefor the calls should now always get
proxied.
Signed-off-by: Tim Marx <t.marx@proxmox.com>
it's a bit strange that one cannot pass the default value explicitly,
helpfull when calling this API path through the CLI envrionment,
which currently cannot have optional fixed-positioned default values
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
As this is now the default behavior in all other ceph api endpoints,
I adapted the status api correspondingly.
We also pass our ceph configuration file directly when connecting to
RADOS, so a /etc/ceph/ceph.conf isn't necessarily required to
indicate a fully setup and enabled PVE-ceph environment.
Signed-off-by: Tim Marx <t.marx@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Instead of opening proc/mounts through IO::File directly for parsing,
the patch uses ProcFSTools. This way it also takes care of eventual
decoding.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
When destroying an OSD over API or CLI, e.g. by executing:
'pveceph osd destroy <num> --cleanup'
all disks associated with the OSD got wiped with dd, which included
any shared and by others still in use ones, e.g., separate disks with
DB/WAL.
The patch changes 'wipe_disks' to wipe the partition instead of the
whole disk.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
over from the time where corosync was still bnased on XML configs
(pre PVE 4.0). This was not used, and XML::Parser is not Export
based, so it does not pushes some methods into the using modules
namespace
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
when using 'check_permissions' directly, we have to actually use the
nodename in the path, else we check the wrong permission and one
needed to have propagating 'Sys.Audit' permissions on '/nodes' for
this to work.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This adds the bash completion for the wakeonlan API call.
The bash completion returns only those nodes which are offline according to the
member status.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
both task sources use PVE::Tools::upid_decode, which ensures all of the
':' delimited fields are set.
this only leaves 'status' and 'endtime' as optional, which are not set
for currently running tasks.
reorder them in the code for easier matching with their on-disk format.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Acked-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
convenience filter if caller is only interested in certain actions
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
otherwise there is no way to find out about (all) active tasks over the
API if their UPIDs were not recorded when the initial API calls happened.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Use the new format to verify the MAC addresses.
The wakeonlan API call now returns the MAC address of the node to wake on
successful sending of the WoL packet.
pvenode finally displays this MAC address to the user as feedback.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Checks if the specified target node even exists before trying to read its MAC
address from the config and sending the wake on LAN magic packet.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Provides the basic functionality to provide a wake on LAN feature implementation
to start nodes in a cluster from other nodes.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
It makes more sense to have it there, especially since we want to
split out the service parts into a seperate file.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Allow to specify a separate cluster network when initializing ceph.
Ceph docs[0] imply a possibility for performance increase and
enhanced security in environments where the public network serves not
fully trusted peers, which could else provoke a DOS to the cluster
traffic[0].
Make this optional, but if passed `network` is required too.
[0]: http://docs.ceph.com/docs/luminous/rados/configuration/network-config-ref/
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
A MDS gets only active once a FS is there, and we need an MDS active
to be able to add a storage, as the CephFS plugin does an immediate
mount check. As an MDS needs some time to get active we had a
problematic time window where this mounting could fail.
Wait for a MDS to get in active state.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Allow to create a new CephFS instance and allow to list them.
As deletion requires coordination between the active MDS and all
standby MDS next in line this needs a bit more work. One could mark
the MDS cluster down and stop the active, that should work but as
destroying is quite a sensible operation, in production not often
needed I deemed it better to document this only, and leaving API
endpoints for this to the future.
For index/list I slightly transform the result of an RADOS `fs ls`
monitor command, this would allow relative easy display of a CephFS
and it's backing metadata and data pools in a GUI.
While for now it's not enabled by default and marked as experimental,
this API is designed to host multiple CephFS instances - we may not
need this at all, but I did not want to limit us early. And anybody
liking to experiment can use it after the respective ceph.conf
settings.
When encountering errors try to rollback. As we verified at the
beginning that we did not reused pools, destroy the ones which we
created.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
Allow to create, list and destroy and Ceph Metadata Server (MDS) over
the API and the CLI `pveceph` tool.
Besides setting up the local systemd service template and the MDS
data directory we also add a reference to the MDS in the ceph.conf
We note the backing host (node) from the respective MDS and set up a
'mds standby for name' = 'pve' so that the PVE created ones are a
single group. If we decide to add integration for rank/path specific
MDS (possible useful for CephFS with quite a bit of load) then this
may help as a starting point.
On create, check early if a reference already exists in ceph.conf and
abort in that case. If we only see existing data directories later
on we abort but do not remove them, they could well be from an older
manual create - where it's possible dangerous to just remove it. Let
the user handle it themself in that case.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
We will reuse this in the future, e.g., when creating a data and
metadata pool for CephFS.
Allow to pass a $rados object (to reuse it, as initializing is not
that cheap) but also create it if it's undefined, fro convenience.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
most of this was imported by just copying without verifying if all is
actually required. Some lost its purpose as we re-used more from our
existing module code base (e.g., pve-common) but wasn't actually
removed.
As this file includes two perl modules you need to take a bit caution
when looking at this, as some things are used in one module but not
the other - simple grep'ing at this may give false positives.
Also add PVE::API2::Storage use which was missing here.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
most of this was imported by just copying without verifying if all is
actually required. Some lost its purpose as we re-used more from our
existing module code base (e.g., pve-common) but wasn't actually
removed.
As this file includes two perl modules you need to take a bit caution
when looking at this, as some things are used in one module but not
the other - simple grep'ing at this may give false positives.
Also include the missing IO::File use.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this will be used for the api endpoints in the future as
PVE::API2::Scan instead of PVE::API2::Storage::Scan since it will
contain endpoints to other modules (like qemu-server for pci/usb
scanning)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This patch removes the separate storage entries for CT & VM to the same
ceph pool. Instead only one entry is made as we can now map/unmap
volumes actively in pve-container.
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
workaround to keep the subscription popup on login even without 'Sys.Audit'
permissions but remove the subscription details in the GUI for unauthorized
users.
this allows the disk to be reused as ceph disk by zeroing the first 200M
of the destroyed disk. disks are iterated separately from partitions to
prevent duplicate wipes.
Signed-off-by: David Limbeck <d.limbeck@proxmox.com>
This add a new api to online reload networking configuration
with ifupdown2.
This work with native ifupdown2 modules, as ifupdown2 have
interface dependency relationships.
Some specific interfaces options can't be reloaded online
(because kernel don't implement it), it this case, we ifdown/ifup
theses interfaces. (mainly vxlan interfaces options)
