Commit Graph

532 Commits

Author SHA1 Message Date
Thomas Lamprecht
017bb1a8bd minor typo fix and code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-02-08 15:26:44 +01:00
Alwin Antreich
3c6aa3f47e api osd/destroy: use ProcFSTools to iterate mounts
Instead of opening proc/mounts through IO::File directly for parsing,
the patch uses ProcFSTools. This way it also takes care of eventual
decoding.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2019-02-08 15:26:00 +01:00
Thomas Lamprecht
e45cc727e6 follouwp: code cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-02-08 15:26:00 +01:00
Alwin Antreich
b436dca874 Fix #2051: preserve DB/WAL disk on destroy
When destroying an OSD over API or CLI, e.g. by executing:

'pveceph osd destroy <num> --cleanup'

all disks associated with the OSD got wiped with dd, which included
any shared and by others still in use ones, e.g., separate disks with
DB/WAL.

The patch changes 'wipe_disks' to wipe the partition instead of the
whole disk.

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-02-08 14:20:23 +01:00
Thomas Lamprecht
dd6b2e5742 remove unused libxml-parser-perl dependency
over from the time where corosync was still bnased on XML configs
(pre PVE 4.0). This was not used, and XML::Parser is not Export
based, so it does not pushes some methods into the using modules
namespace

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-01-31 12:02:55 +01:00
Thomas Lamprecht
7d762f4cc0 cleanup: api/subscription: keep variable declarations closer to use
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-01-31 10:54:50 +01:00
Dominik Csapak
7d6fba8fff api/subscription: fix permission check
when using 'check_permissions' directly, we have to actually use the
nodename in the path, else we check the wrong permission and one
needed to have propagating 'Sys.Audit' permissions on '/nodes' for
this to work.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2019-01-31 10:52:38 +01:00
Christian Ebner
5aa7b909b1 Add bash completion for wakeonlan API call
This adds the bash completion for the wakeonlan API call.
The bash completion returns only those nodes which are offline according to the
member status.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-01-28 17:28:34 +01:00
Fabian Grünbichler
f6faf09cbf task index: fix return schema annotations
both task sources use PVE::Tools::upid_decode, which ensures all of the
':' delimited fields are set.

this only leaves 'status' and 'endtime' as optional, which are not set
for currently running tasks.

reorder them in the code for easier matching with their on-disk format.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Acked-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-01-23 18:42:59 +01:00
Fabian Grünbichler
8dacbee3e2 task index: allow filtering by task type
convenience filter if caller is only interested in certain actions

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-01-23 18:42:59 +01:00
Fabian Grünbichler
32388c4128 task index: allow selection of task source(s)
otherwise there is no way to find out about (all) active tasks over the
API if their UPIDs were not recorded when the initial API calls happened.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-01-23 18:42:59 +01:00
Thomas Lamprecht
ec1788043c api/wakeonlan: add description to return schema
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-01-22 12:19:07 +01:00
Christian Ebner
0f615ea9df pvenode: Return MAC address used for WoL call on success
Use the new format to verify the MAC addresses.
The wakeonlan API call now returns the MAC address of the node to wake on
successful sending of the WoL packet.
pvenode finally displays this MAC address to the user as feedback.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-01-22 12:05:47 +01:00
Thomas Lamprecht
824369963b wakeonlan: we cannot wake ourself
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-01-21 16:49:34 +01:00
Christian Ebner
6f712496d8 Nodes: Check if target node even exists before sending WOL packet
Checks if the specified target node even exists before trying to read its MAC
address from the config and sending the wake on LAN magic packet.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-01-21 16:22:12 +01:00
Christian Ebner
b3d8454221 implement basic wake on LAN functionality
Provides the basic functionality to provide a wake on LAN feature implementation
to start nodes in a cluster from other nodes.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-01-17 11:15:01 +01:00
Thomas Lamprecht
9fdfebf743 allow 8-way NUMA systems
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-01-11 10:21:55 +01:00
Fabian Grünbichler
742d2ad20e fix #1997: set taskid for single-guest backups
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2019-01-03 09:17:38 +01:00
Dominik Csapak
98fe93ae25 ceph: move Monitor API calls to API2/Ceph/MON.pm
and adapt the paths

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-12-20 09:44:01 +01:00
Dominik Csapak
4fec2764f1 ceph: move MGR API calls to API2/Ceph/MGR.pm
and adapt the paths

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-12-20 09:44:01 +01:00
Dominik Csapak
be7edba15d ceph: move mgr create/destroy to Ceph::Services
and adapt the paths and callers

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-12-20 09:44:01 +01:00
Dominik Csapak
79fa41a2b8 ceph: move API2::CephOSD to API2::Ceph::OSD
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-12-20 09:44:01 +01:00
Dominik Csapak
27439be616 ceph: move service_cmd and MDS related code to Services.pm
Also adapts the calls to the relevant subs.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-12-20 09:44:01 +01:00
Dominik Csapak
6fb08cb923 ceph: move CephTools into Ceph/Tools.pm
It makes more sense to have it there, especially since we want to
split out the service parts into a seperate file.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-12-20 09:44:01 +01:00
Dominik Csapak
c31f487e7a ceph: use cfs_read/write_file for ceph.conf
The parser is now registered, and ceph.conf is a tracked file in pmxcfs.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-12-20 09:44:01 +01:00
Thomas Lamprecht
c56d75b4b6 fixup: whitespace cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-12-13 10:15:46 +01:00
Tim Marx
1aa902ae90 ceph api: added check for /etc/pve/ceph.conf to remaining/new endpoints
Signed-off-by: Tim Marx <t.marx@proxmox.com>
2018-12-13 10:14:39 +01:00
Thomas Lamprecht
6ad70a2bb8 ceph: update all pg_num defaults to 128
As the last patch missed a few places. see pg calc or also:
http://docs.ceph.com/docs/luminous/rados/operations/placement-groups/

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-29 12:36:25 +01:00
Alwin Antreich
ba454377ad Raise the default pg_num to 128
PGCalc and the ceph recommendation say 128 PGs for up to 5 OSDs

Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-11-29 12:19:22 +01:00
Thomas Lamprecht
1343ae6db5 fixup: s/wipe_disk/wipe_disks/ as we can to multiple at once
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-28 10:30:58 +01:00
Alwin Antreich
456a7f4d34 Factoring the wipe_disk out into CephTools
Signed-off-by: Alwin Antreich <a.antreich@proxmox.com>
2018-11-28 10:25:16 +01:00
Thomas Lamprecht
c5a0a1e449 ceph: fixup s/add_storage/add-storage/
it's just the nicer interface and we want to go away from underscore
usage in new calls

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-28 10:07:08 +01:00
Thomas Lamprecht
7519b84851 fix #1430: ceph init: allow to specify separate cluster network
Allow to specify a separate cluster network when initializing ceph.
Ceph docs[0] imply a possibility for performance increase and
enhanced security in environments where the public network serves not
fully trusted peers, which could else provoke a DOS to the cluster
traffic[0].

Make this optional, but if passed `network` is required too.

[0]: http://docs.ceph.com/docs/luminous/rados/configuration/network-config-ref/

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-28 09:29:22 +01:00
Thomas Lamprecht
e337caaf46 api: cephfs: reuse rados connection when polling for active MDS
no point in recreating one, we have an active one from earlier

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-27 15:24:08 +01:00
Thomas Lamprecht
a62d7bd966 api: cephfs: wait for MDS to become active
A MDS gets only active once a FS is there, and we need an MDS active
to be able to add a storage, as the CephFS plugin does an immediate
mount check. As an MDS needs some time to get active we had a
problematic time window where this mounting could fail.

Wait for a MDS to get in active state.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-23 18:49:48 +01:00
Thomas Lamprecht
34c1236c35 api: cephfs: check if SID is free when add_storage is set
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-23 18:35:16 +01:00
Thomas Lamprecht
97fe40b0be api/ceph: import file_set_contents
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-23 16:17:13 +01:00
Thomas Lamprecht
7e1a9d25b6 ceph: add CephFS create and list API
Allow to create a new CephFS instance and allow to list them.

As deletion requires coordination between the active MDS and all
standby MDS next in line this needs a bit more work. One could mark
the MDS cluster down and stop the active, that should work but as
destroying is quite a sensible operation, in production not often
needed I deemed it better to document this only, and leaving API
endpoints for this to the future.

For index/list I slightly transform the result of an RADOS `fs ls`
monitor command, this would allow relative easy display of a CephFS
and it's backing metadata and data pools in a GUI.

While for now it's not enabled by default and marked as experimental,
this API is designed to host multiple CephFS instances - we may not
need this at all, but I did not want to limit us early. And anybody
liking to experiment can use it after the respective ceph.conf
settings.

When encountering errors try to rollback. As we verified at the
beginning that we did not reused pools, destroy the ones which we
created.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
2018-11-23 13:33:12 +01:00
Thomas Lamprecht
b82649cc52 ceph: add MDS create/delete/list API
Allow to create, list and destroy and Ceph Metadata Server (MDS) over
the API and the CLI `pveceph` tool.

Besides setting up the local systemd service template and the MDS
data directory we also add a reference to the MDS in the ceph.conf
We note the backing host (node) from the respective MDS and set up a
'mds standby for name' = 'pve' so that the PVE created ones are a
single group. If we decide to add integration for rank/path specific
MDS (possible useful for CephFS with quite a bit of load) then this
may help as a starting point.

On create, check early if a reference already exists in ceph.conf and
abort in that case. If we only see existing data directories later
on we abort but do not remove them, they could well be from an older
manual create - where it's possible dangerous to just remove it. Let
the user handle it themself in that case.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Co-authored-by: Alwin Antreich <a.antreich@proxmox.com>
2018-11-23 13:33:12 +01:00
Thomas Lamprecht
f96d701292 ceph: move create/destroy pool to CephTools
We will reuse this in the future, e.g., when creating a data and
metadata pool for CephFS.

Allow to pass a $rados object (to reuse it, as initializing is not
that cheap) but also create it if it's undefined, fro convenience.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-23 13:33:12 +01:00
Thomas Lamprecht
33a9c70a45 api: document default for ceph service start/stop/restart
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-23 12:40:46 +01:00
Dominik Csapak
342c0830cf ceph: add service restart api call
like start/stop but for restart, this makes maintenance a bit easier

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-11-23 12:23:42 +01:00
Thomas Lamprecht
3f2300d8b4 api/hardware: fixup: add missing file
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-22 15:05:52 +01:00
Thomas Lamprecht
523d5f486b api: move PCI from Scan to Hardware path
as discussed[0] lets move this to /nodes/{node}/hardware/pci

Put this in its own subfolder and adapt module names and paths in UI.

[0]: https://pve.proxmox.com/pipermail/pve-devel/2018-November/034694.html
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-22 14:20:09 +01:00
Dominik Csapak
07d982de6c api/Scan: allow to get a PCI device's mediated devices
this is for the gui to be able to select mediated devices
path ist /node/NODE/scan/pci/PCIID/mdev

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-11-22 13:30:50 +01:00
Dominik Csapak
782be7dae2 api/Scan: allow to scan a nodes PCI devices
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-11-22 13:30:21 +01:00
Thomas Lamprecht
400742e432 api/Ceph: actually use imported file_{g,s}et_contents 2018-11-22 11:06:12 +01:00
Thomas Lamprecht
0cd34b008b api/Ceph: cleanup module usage
most of this was imported by just copying without verifying if all is
actually required. Some lost its purpose as we re-used more from our
existing module code base (e.g., pve-common) but wasn't actually
removed.

As this file includes two perl modules you need to take a bit caution
when looking at this, as some things are used in one module but not
the other - simple grep'ing at this may give false positives.

Also add PVE::API2::Storage use which was missing here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-22 11:06:12 +01:00
Thomas Lamprecht
d4d2e789b8 api/CephOSD: cleanup and sort module usage
most of this was imported by just copying without verifying if all is
actually required. Some lost its purpose as we re-used more from our
existing module code base (e.g., pve-common) but wasn't actually
removed.

As this file includes two perl modules you need to take a bit caution
when looking at this, as some things are used in one module but not
the other - simple grep'ing at this may give false positives.

Also include the missing IO::File use.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2018-11-22 11:06:12 +01:00
Dominik Csapak
ab322f5f83 rename PVE::API2::Storage::Scan to PVE::API2::Scan
and use it in the API

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2018-11-19 15:04:31 +01:00