mirror of
https://git.proxmox.com/git/mirror_corosync
synced 2025-10-15 06:51:25 +00:00
votequorum: major rework to fix qdevice API and integration with core
qdevice is a very special node in the cluster and it adds a certain
amount of complexity and special cases across the code.
most of the qdevice data are shared across the cluster (name/votes)
but effectively each node has a different view of the qdevice
(registered/unregistered/voting/etc.)
with this change, we align the qdevice view across the node,
exchanging more data between nodes and we fix how qdevice behaves
and it is configured.
The only side effect is that the amount of data transmitted on wire
is slightly higher.
The qdevice API is still disabled by default. This means that
the amount of real changes in current code are a lot smaller
than it appears by this patch.
TODO: documentation/man pages needs to be updated once
this change is in (and behavior finalized).
User visible changes:
- configuration (coroparse, exec/votequorum):
the quorum device section is now standalone within the quorum.
quorum {
provider: corosync_votequorum
device {
model: (name)
timeout: (millisec)
votes:
}
}
the keyword "model:" is mandatory to enable qdevice in configuration
and should express the name of the script/daemon that will provide
the qdevice. Looking into the future, an init script or systemd
service will look for that name in /path/to/be/decided/name
and start/stop qdevice.
timeout: defines the maximum interval the qdevice implementation
has available between poll (see votequorum_qdevice_poll.3) before
the device is considered dead and votes discarded
votes: is now a configuration parameter and not an API call.
quorum devices don't care what they need to vote.
votes is autocalculated when a nodelist is available and all
nodes in the list vote 1. Otherwise this parameter is mandatory.
- configuration (exec/votequorum):
startup and runtime configuration changes have been improved.
errors at startup are considered fatal. errors at runtime
have different exit paths.
startup:
* quorum.two_node and qdevice are incompatible.
* quorum.expected_votes requires quorum.device.votes.
* quorum.expected_votes - quorum.device.votes cannot be lower
than 2.
* qdevice and last_man_standing are mutually exclusive.
* qdevice and auto_tie_breaker are mutually exclusive.
runtime config changes:
* quorum.two_node and qdevice are incompatible:
if quorum device is alive, two_node is disabled.
if quorum device is not alive and node count is 2, two_node is
enabled, and quorum device cannot be registered
* if either last_man_standing or auto_tie_breaker were enabled
at startup, and at runtime quorum device is configured,
quorum device registration will be blocked.
* if quorum.expected_votes is configured but not quorum.device.votes,
quorum device registration will be blocked.
* if quorum.device.votes is not configured and we cannot
automatically calculate it, quorum device registration will be blocked.
* An error in configuring quorum.expected_votes and quorum.device.votes
will block quorum device registration.
blocking quorum device registation, also means dropping the votes.
quorum.device.votes (either set or automatically calculated) is now
used to determine current expected_votes in the cluster.
- logging (exec/votequorum):
all errors from configuration are treated as WARNING/CRITICAL.
lots of extra DEBUG output is added (see internal changes too).
- corosync-quorumtool (tools/corosync-quorumtool):
* added option to forcefully kick out a quorum device from the local
node. This is for emergency recovery only and it is only
available when qdevice API is built-in.
* Improved status output, specifically add node state and qdevice
information
[root@fedora-master-node2 coro]# corosync-quorumtool -s
Version: 1.99.4.12-9c7d-dirty
Quorum type: corosync_votequorum
Nodes: 2
Ring ID: 132
Quorate: Yes
Node votes: 1
Node state: Member
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate Qdevice
Nodeid Votes Name
1 1 fedora-master-node1.int.fabbione.net
2 1 fedora-master-node2.int.fabbione.net
0 1 QDEVICE (Voting)
* allow to print status for any node in the cluster known to
local node.
[root@fedora-master-node1 coro]# corosync-quorumtool -s
Version: 1.99.4.12-9c7d-dirty
Quorum type: corosync_votequorum
Nodes: 2
Ring ID: 144
Quorate: Yes
Node votes: 1
Node state: Member
Expected votes: 3
Highest expected: 3
Total votes: 2
Quorum: 2
Flags: Quorate
Nodeid Votes Name
1 1 fedora-master-node1.int.fabbione.net
2 1 fedora-master-node2.int.fabbione.net
[root@fedora-master-node1 coro]# corosync-quorumtool -s -n 2
Version: 1.99.4.12-9c7d-dirty
Quorum type: corosync_votequorum
Nodes: 2
Ring ID: 144
Quorate: Yes
Node votes: 1
Node state: Member
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate Qdevice
Nodeid Votes Name
1 1 fedora-master-node1.int.fabbione.net
2 1 fedora-master-node2.int.fabbione.net
0 1 QDEVICE (Voting)
Internal changes:
- change qdevice timer to not run all time, but only when necessary.
- change votequorum_nodeinfo on wire data to use flags instead of uint8_t
and add QDEVICE status.
- allocate nodeid 0 to qdevice since it's the only real
nodeid that be reserved.
- change send_nodeinfo to allow to send nodeinfo for any node
so that we can share qdevice info across the cluster
(and this might be useful in future if we need to sync
internal cluster view).
- add votequorum api call to update qdevice name
- add runtime data if quorum device has been forcefully disabled
by config error
- add qdevice votes to expected_votes calculation (this
is probably the biggest difference vs cman)
- change votequorum_read_nodelist_configuration so that
we can autocalculate votes for qdevice (we need the nodecount
vs votes).
- add all checks for startup/runtime config (see above).
- do not make qdevice part of the membership_list received from
totem. None of our users care about it and it is not a real node.
- change onwire message handlers to deal with "data for this node from any node"
case and undersand nodeid 0 for qdevice info
- always allocate qdevice at startup. this simplifies code a lot.
- dispatch qdevice nodeinfo on membership changes.
- inform libvotequorum users when a qdevice is registered
- improve substantially qdevice api and add a simple
barrier based on qdevice name.
- add qdevice API barrier at cluster level. This feature allow
only one qdevice name to be active in the cluster at any time.
- qdevice getinfo can now report status for qdevice on any node.
- change slightly the way the qdevice API is built-in/out:
only the libvotequorum calls are #ifdef'out now. Doing so in
the core is too complex and would make the code unreadable
with the risk of missing a bit or two effectively introducing
an on-wire incompatibility if we will ever turn the API on.
- probably added some bugs on the way...
TODO: update qdevice_* API once the above is settled and test
qdevice integration with other features.
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com> (only second part)
This commit is contained in:
Fabio M. Di Nitto
parent
9c7d1d3096
commit
cb5fd77501
@ -87,6 +87,7 @@ enum main_cp_cb_data_state {
|
||||
MAIN_CP_CB_DATA_STATE_LOGGING_DAEMON,
|
||||
MAIN_CP_CB_DATA_STATE_MEMBER,
|
||||
MAIN_CP_CB_DATA_STATE_QUORUM,
|
||||
MAIN_CP_CB_DATA_STATE_QDEVICE,
|
||||
MAIN_CP_CB_DATA_STATE_NODELIST,
|
||||
MAIN_CP_CB_DATA_STATE_NODELIST_NODE,
|
||||
};
|
||||
@ -388,7 +389,6 @@ static int main_config_parser_cb(const char *path,
|
||||
case MAIN_CP_CB_DATA_STATE_QUORUM:
|
||||
if ((strcmp(path, "quorum.expected_votes") == 0) ||
|
||||
(strcmp(path, "quorum.votes") == 0) ||
|
||||
(strcmp(path, "quorum.quorumdev_poll") == 0) ||
|
||||
(strcmp(path, "quorum.last_man_standing_window") == 0) ||
|
||||
(strcmp(path, "quorum.leaving_timeout") == 0)) {
|
||||
if (safe_atoi(value, &i) != 0) {
|
||||
@ -410,6 +410,15 @@ static int main_config_parser_cb(const char *path,
|
||||
add_as_string = 0;
|
||||
}
|
||||
break;
|
||||
case MAIN_CP_CB_DATA_STATE_QDEVICE:
|
||||
if ((strcmp(path, "quorum.device.timeout") == 0) ||
|
||||
(strcmp(path, "quorum.device.votes") == 0)) {
|
||||
if (safe_atoi(value, &i) != 0) {
|
||||
goto atoi_error;
|
||||
}
|
||||
icmap_set_uint32(path, i);
|
||||
add_as_string = 0;
|
||||
}
|
||||
case MAIN_CP_CB_DATA_STATE_TOTEM:
|
||||
if ((strcmp(path, "totem.version") == 0) ||
|
||||
(strcmp(path, "totem.nodeid") == 0) ||
|
||||
@ -671,6 +680,9 @@ static int main_config_parser_cb(const char *path,
|
||||
if (strcmp(path, "quorum") == 0) {
|
||||
data->state = MAIN_CP_CB_DATA_STATE_QUORUM;
|
||||
}
|
||||
if (strcmp(path, "quorum.device") == 0) {
|
||||
data->state = MAIN_CP_CB_DATA_STATE_QDEVICE;
|
||||
}
|
||||
if (strcmp(path, "nodelist") == 0) {
|
||||
data->state = MAIN_CP_CB_DATA_STATE_NODELIST;
|
||||
data->node_number = 0;
|
||||
@ -858,6 +870,9 @@ static int main_config_parser_cb(const char *path,
|
||||
case MAIN_CP_CB_DATA_STATE_QUORUM:
|
||||
data->state = MAIN_CP_CB_DATA_STATE_NORMAL;
|
||||
break;
|
||||
case MAIN_CP_CB_DATA_STATE_QDEVICE:
|
||||
data->state = MAIN_CP_CB_DATA_STATE_QUORUM;
|
||||
break;
|
||||
case MAIN_CP_CB_DATA_STATE_NODELIST:
|
||||
data->state = MAIN_CP_CB_DATA_STATE_NORMAL;
|
||||
break;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -48,6 +48,7 @@ enum req_votequorum_types {
|
||||
,
|
||||
MESSAGE_REQ_VOTEQUORUM_QDEVICE_REGISTER,
|
||||
MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER,
|
||||
MESSAGE_REQ_VOTEQUORUM_QDEVICE_UPDATE,
|
||||
MESSAGE_REQ_VOTEQUORUM_QDEVICE_POLL,
|
||||
MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO
|
||||
#endif
|
||||
@ -62,23 +63,39 @@ enum res_votequorum_types {
|
||||
MESSAGE_RES_VOTEQUORUM_EXPECTEDVOTES_NOTIFICATION
|
||||
};
|
||||
|
||||
struct req_lib_votequorum_setvotes {
|
||||
struct qb_ipc_request_header header __attribute__((aligned(8)));
|
||||
unsigned int votes;
|
||||
int nodeid;
|
||||
};
|
||||
|
||||
struct req_lib_votequorum_qdevice_register {
|
||||
struct qb_ipc_request_header header __attribute__((aligned(8)));
|
||||
unsigned int votes;
|
||||
char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
|
||||
};
|
||||
|
||||
struct req_lib_votequorum_qdevice_unregister {
|
||||
struct qb_ipc_request_header header __attribute__((aligned(8)));
|
||||
char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
|
||||
};
|
||||
|
||||
struct req_lib_votequorum_qdevice_update {
|
||||
struct qb_ipc_request_header header __attribute__((aligned(8)));
|
||||
char oldname[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
|
||||
char newname[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
|
||||
};
|
||||
|
||||
struct req_lib_votequorum_qdevice_poll {
|
||||
struct qb_ipc_request_header header __attribute__((aligned(8)));
|
||||
char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
|
||||
int state;
|
||||
};
|
||||
|
||||
struct req_lib_votequorum_qdevice_getinfo {
|
||||
struct qb_ipc_request_header header __attribute__((aligned(8)));
|
||||
int nodeid;
|
||||
};
|
||||
|
||||
struct req_lib_votequorum_setvotes {
|
||||
struct qb_ipc_request_header header __attribute__((aligned(8)));
|
||||
unsigned int votes;
|
||||
int nodeid;
|
||||
};
|
||||
|
||||
struct req_lib_votequorum_setexpected {
|
||||
struct qb_ipc_request_header header __attribute__((aligned(8)));
|
||||
unsigned int expected_votes;
|
||||
@ -109,10 +126,12 @@ struct res_lib_votequorum_status {
|
||||
#define VOTEQUORUM_INFO_LAST_MAN_STANDING 8
|
||||
#define VOTEQUORUM_INFO_AUTO_TIE_BREAKER 16
|
||||
#define VOTEQUORUM_INFO_LEAVE_REMOVE 32
|
||||
#define VOTEQUORUM_INFO_QDEVICE 64
|
||||
|
||||
struct res_lib_votequorum_getinfo {
|
||||
struct qb_ipc_response_header header __attribute__((aligned(8)));
|
||||
int nodeid;
|
||||
unsigned int nodeid;
|
||||
unsigned int state;
|
||||
unsigned int votes;
|
||||
unsigned int expected_votes;
|
||||
unsigned int highest_expected;
|
||||
|
||||
@ -42,19 +42,16 @@ extern "C" {
|
||||
|
||||
typedef uint64_t votequorum_handle_t;
|
||||
|
||||
#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
|
||||
#define VOTEQUORUM_MAX_QDEVICE_NAME_LEN 255
|
||||
#endif
|
||||
|
||||
#define VOTEQUORUM_INFO_FLAG_TWONODE 1
|
||||
#define VOTEQUORUM_INFO_FLAG_QUORATE 2
|
||||
#define VOTEQUORUM_INFO_WAIT_FOR_ALL 4
|
||||
#define VOTEQUORUM_INFO_LAST_MAN_STANDING 8
|
||||
#define VOTEQUORUM_INFO_AUTO_TIE_BREAKER 16
|
||||
#define VOTEQUORUM_INFO_LEAVE_REMOVE 32
|
||||
#define VOTEQUORUM_INFO_QDEVICE 64
|
||||
|
||||
#define VOTEQUORUM_NODEID_US 0
|
||||
#define VOTEQUORUM_NODEID_QDEVICE -1
|
||||
#define VOTEQUORUM_NODEID_QDEVICE 0
|
||||
#define VOTEQUORUM_MAX_QDEVICE_NAME_LEN 255
|
||||
|
||||
#define NODESTATE_MEMBER 1
|
||||
#define NODESTATE_DEAD 2
|
||||
@ -64,6 +61,7 @@ typedef uint64_t votequorum_handle_t;
|
||||
|
||||
struct votequorum_info {
|
||||
unsigned int node_id;
|
||||
unsigned int state;
|
||||
unsigned int node_votes;
|
||||
unsigned int node_expected_votes;
|
||||
unsigned int highest_expected;
|
||||
@ -187,20 +185,29 @@ cs_error_t votequorum_context_set (
|
||||
*/
|
||||
cs_error_t votequorum_qdevice_register (
|
||||
votequorum_handle_t handle,
|
||||
const char *name,
|
||||
unsigned int votes);
|
||||
const char *name);
|
||||
|
||||
/**
|
||||
* Unregister a quorum device
|
||||
*/
|
||||
cs_error_t votequorum_qdevice_unregister (
|
||||
votequorum_handle_t handle);
|
||||
votequorum_handle_t handle,
|
||||
const char *name);
|
||||
|
||||
/**
|
||||
* Update registered name of a quorum device
|
||||
*/
|
||||
cs_error_t votequorum_qdevice_update (
|
||||
votequorum_handle_t handle,
|
||||
const char *oldname,
|
||||
const char *newname);
|
||||
|
||||
/**
|
||||
* Poll a quorum device
|
||||
*/
|
||||
cs_error_t votequorum_qdevice_poll (
|
||||
votequorum_handle_t handle,
|
||||
const char *name,
|
||||
unsigned int state);
|
||||
|
||||
/**
|
||||
@ -208,6 +215,7 @@ cs_error_t votequorum_qdevice_poll (
|
||||
*/
|
||||
cs_error_t votequorum_qdevice_getinfo (
|
||||
votequorum_handle_t handle,
|
||||
unsigned int nodeid,
|
||||
struct votequorum_qdevice_info *info);
|
||||
|
||||
#endif
|
||||
|
||||
@ -174,6 +174,7 @@ cs_error_t votequorum_getinfo (
|
||||
error = res_lib_votequorum_getinfo.header.error;
|
||||
|
||||
info->node_id = res_lib_votequorum_getinfo.nodeid;
|
||||
info->state = res_lib_votequorum_getinfo.state;
|
||||
info->node_votes = res_lib_votequorum_getinfo.votes;
|
||||
info->node_expected_votes = res_lib_votequorum_getinfo.expected_votes;
|
||||
info->highest_expected = res_lib_votequorum_getinfo.highest_expected;
|
||||
@ -539,8 +540,7 @@ error_put:
|
||||
#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
|
||||
cs_error_t votequorum_qdevice_register (
|
||||
votequorum_handle_t handle,
|
||||
const char *name,
|
||||
unsigned int votes)
|
||||
const char *name)
|
||||
{
|
||||
cs_error_t error;
|
||||
struct votequorum_inst *votequorum_inst;
|
||||
@ -548,8 +548,9 @@ cs_error_t votequorum_qdevice_register (
|
||||
struct req_lib_votequorum_qdevice_register req_lib_votequorum_qdevice_register;
|
||||
struct res_lib_votequorum_status res_lib_votequorum_status;
|
||||
|
||||
if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN)
|
||||
if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
|
||||
return CS_ERR_INVALID_PARAM;
|
||||
}
|
||||
|
||||
error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
|
||||
if (error != CS_OK) {
|
||||
@ -560,7 +561,6 @@ cs_error_t votequorum_qdevice_register (
|
||||
req_lib_votequorum_qdevice_register.header.size = sizeof (struct req_lib_votequorum_qdevice_register);
|
||||
req_lib_votequorum_qdevice_register.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_REGISTER;
|
||||
strcpy(req_lib_votequorum_qdevice_register.name, name);
|
||||
req_lib_votequorum_qdevice_register.votes = votes;
|
||||
|
||||
iov.iov_base = (char *)&req_lib_votequorum_qdevice_register;
|
||||
iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_register);
|
||||
@ -586,6 +586,7 @@ error_exit:
|
||||
|
||||
cs_error_t votequorum_qdevice_poll (
|
||||
votequorum_handle_t handle,
|
||||
const char *name,
|
||||
unsigned int state)
|
||||
{
|
||||
cs_error_t error;
|
||||
@ -594,14 +595,18 @@ cs_error_t votequorum_qdevice_poll (
|
||||
struct req_lib_votequorum_qdevice_poll req_lib_votequorum_qdevice_poll;
|
||||
struct res_lib_votequorum_status res_lib_votequorum_status;
|
||||
|
||||
if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
|
||||
return CS_ERR_INVALID_PARAM;
|
||||
}
|
||||
|
||||
error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
|
||||
if (error != CS_OK) {
|
||||
return (error);
|
||||
}
|
||||
|
||||
|
||||
req_lib_votequorum_qdevice_poll.header.size = sizeof (struct req_lib_votequorum_qdevice_poll);
|
||||
req_lib_votequorum_qdevice_poll.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_POLL;
|
||||
strcpy(req_lib_votequorum_qdevice_poll.name, name);
|
||||
req_lib_votequorum_qdevice_poll.state = state;
|
||||
|
||||
iov.iov_base = (char *)&req_lib_votequorum_qdevice_poll;
|
||||
@ -627,24 +632,30 @@ error_exit:
|
||||
}
|
||||
|
||||
cs_error_t votequorum_qdevice_unregister (
|
||||
votequorum_handle_t handle)
|
||||
votequorum_handle_t handle,
|
||||
const char *name)
|
||||
{
|
||||
cs_error_t error;
|
||||
struct votequorum_inst *votequorum_inst;
|
||||
struct iovec iov;
|
||||
struct req_lib_votequorum_general req_lib_votequorum_general;
|
||||
struct req_lib_votequorum_qdevice_unregister req_lib_votequorum_qdevice_unregister;
|
||||
struct res_lib_votequorum_status res_lib_votequorum_status;
|
||||
|
||||
if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
|
||||
return CS_ERR_INVALID_PARAM;
|
||||
}
|
||||
|
||||
error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
|
||||
if (error != CS_OK) {
|
||||
return (error);
|
||||
}
|
||||
|
||||
req_lib_votequorum_general.header.size = sizeof (struct req_lib_votequorum_general);
|
||||
req_lib_votequorum_general.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER;
|
||||
req_lib_votequorum_qdevice_unregister.header.size = sizeof (struct req_lib_votequorum_qdevice_unregister);
|
||||
req_lib_votequorum_qdevice_unregister.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER;
|
||||
strcpy(req_lib_votequorum_qdevice_unregister.name, name);
|
||||
|
||||
iov.iov_base = (char *)&req_lib_votequorum_general;
|
||||
iov.iov_len = sizeof (struct req_lib_votequorum_general);
|
||||
iov.iov_base = (char *)&req_lib_votequorum_qdevice_unregister;
|
||||
iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_unregister);
|
||||
|
||||
error = qb_to_cs_error(qb_ipcc_sendv_recv (
|
||||
votequorum_inst->c,
|
||||
@ -667,12 +678,13 @@ error_exit:
|
||||
|
||||
cs_error_t votequorum_qdevice_getinfo (
|
||||
votequorum_handle_t handle,
|
||||
unsigned int nodeid,
|
||||
struct votequorum_qdevice_info *qinfo)
|
||||
{
|
||||
cs_error_t error;
|
||||
struct votequorum_inst *votequorum_inst;
|
||||
struct iovec iov;
|
||||
struct req_lib_votequorum_general req_lib_votequorum_general;
|
||||
struct req_lib_votequorum_qdevice_getinfo req_lib_votequorum_qdevice_getinfo;
|
||||
struct res_lib_votequorum_qdevice_getinfo res_lib_votequorum_qdevice_getinfo;
|
||||
|
||||
error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
|
||||
@ -680,12 +692,12 @@ cs_error_t votequorum_qdevice_getinfo (
|
||||
return (error);
|
||||
}
|
||||
|
||||
req_lib_votequorum_qdevice_getinfo.header.size = sizeof (struct req_lib_votequorum_qdevice_getinfo);
|
||||
req_lib_votequorum_qdevice_getinfo.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO;
|
||||
req_lib_votequorum_qdevice_getinfo.nodeid = nodeid;
|
||||
|
||||
req_lib_votequorum_general.header.size = sizeof (struct req_lib_votequorum_general);
|
||||
req_lib_votequorum_general.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO;
|
||||
|
||||
iov.iov_base = (char *)&req_lib_votequorum_general;
|
||||
iov.iov_len = sizeof (struct req_lib_votequorum_general);
|
||||
iov.iov_base = (char *)&req_lib_votequorum_qdevice_getinfo;
|
||||
iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_getinfo);
|
||||
|
||||
error = qb_to_cs_error(qb_ipcc_sendv_recv (
|
||||
votequorum_inst->c,
|
||||
|
||||
@ -50,7 +50,7 @@ static void print_info(int ok_to_fail)
|
||||
struct votequorum_qdevice_info qinfo;
|
||||
int err;
|
||||
|
||||
if ( (err=votequorum_qdevice_getinfo(handle, &qinfo)) != CS_OK)
|
||||
if ( (err=votequorum_qdevice_getinfo(handle, VOTEQUORUM_NODEID_QDEVICE, &qinfo)) != CS_OK)
|
||||
fprintf(stderr, "votequorum_qdevice_getinfo error %d: %s\n", err, ok_to_fail?"OK":"FAILED");
|
||||
else {
|
||||
printf("qdevice votes %d\n", qinfo.votes);
|
||||
@ -63,6 +63,7 @@ static void print_info(int ok_to_fail)
|
||||
|
||||
int main(int argc, char *argv[])
|
||||
{
|
||||
int ret = 0;
|
||||
#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
|
||||
int pollcount=0, polltime=1;
|
||||
int err;
|
||||
@ -82,24 +83,34 @@ int main(int argc, char *argv[])
|
||||
}
|
||||
|
||||
if (argc >= 2) {
|
||||
if ( (err=votequorum_qdevice_register(handle, "QDEVICE", 4)) != CS_OK)
|
||||
if ( (err=votequorum_qdevice_register(handle, "QDEVICE")) != CS_OK) {
|
||||
fprintf(stderr, "qdevice_register FAILED: %d\n", err);
|
||||
ret = -1;
|
||||
goto out;
|
||||
}
|
||||
|
||||
while (pollcount--) {
|
||||
print_info(0);
|
||||
if ((err=votequorum_qdevice_poll(handle, 1)) != CS_OK)
|
||||
if ((err=votequorum_qdevice_poll(handle, "QDEVICE", 1)) != CS_OK) {
|
||||
fprintf(stderr, "qdevice poll FAILED: %d\n", err);
|
||||
ret = -1;
|
||||
goto out;
|
||||
}
|
||||
print_info(0);
|
||||
sleep(polltime);
|
||||
}
|
||||
if ((err= votequorum_qdevice_unregister(handle)) != CS_OK)
|
||||
if ((err= votequorum_qdevice_unregister(handle, "QDEVICE")) != CS_OK) {
|
||||
fprintf(stderr, "qdevice unregister FAILED: %d\n", err);
|
||||
ret = -1;
|
||||
goto out;
|
||||
}
|
||||
}
|
||||
print_info(1);
|
||||
|
||||
out:
|
||||
votequorum_finalize(handle);
|
||||
#else
|
||||
fprintf(stderr, "qdevice support is not built in corosync/votequorum\n");
|
||||
fprintf(stderr, "qdevice support is not built in corosync/votequorum\n");
|
||||
#endif
|
||||
return 0;
|
||||
return ret;
|
||||
}
|
||||
|
||||
@ -63,7 +63,8 @@ typedef enum {
|
||||
CMD_SHOWSTATUS,
|
||||
CMD_SETVOTES,
|
||||
CMD_SETEXPECTED,
|
||||
CMD_MONITOR
|
||||
CMD_MONITOR,
|
||||
CMD_UNREGISTER_QDEVICE
|
||||
} command_t;
|
||||
|
||||
/*
|
||||
@ -108,6 +109,8 @@ static votequorum_callbacks_t v_callbacks = {
|
||||
.votequorum_notify_fn = NULL,
|
||||
.votequorum_expectedvotes_notify_fn = NULL
|
||||
};
|
||||
static uint32_t our_nodeid = 0;
|
||||
static uint8_t display_qdevice = 0;
|
||||
|
||||
/*
|
||||
* cfg bits
|
||||
@ -132,6 +135,7 @@ static void show_usage(const char *name)
|
||||
printf(" -e <expected> change expected votes for the cluster *\n");
|
||||
printf(" -H show nodeids in hexadecimal rather than decimal\n");
|
||||
printf(" -i show node IP addresses instead of the resolved name\n");
|
||||
printf(" -f forcefully unregister a quorum device *DANGEROUS*\n");
|
||||
printf(" -h show this help text\n");
|
||||
printf("\n");
|
||||
printf(" * Starred items only work if votequorum is the quorum provider for corosync\n");
|
||||
@ -283,7 +287,7 @@ static void quorum_notification_fn(
|
||||
}
|
||||
}
|
||||
|
||||
static void display_nodes_data(nodeid_format_t nodeid_format, name_format_t name_format)
|
||||
static void display_nodes_data(uint32_t nodeid, nodeid_format_t nodeid_format, name_format_t name_format)
|
||||
{
|
||||
int i;
|
||||
|
||||
@ -305,13 +309,51 @@ static void display_nodes_data(nodeid_format_t nodeid_format, name_format_t name
|
||||
printf("%s\n", node_name(g_view_list[i], name_format));
|
||||
}
|
||||
}
|
||||
|
||||
if (g_view_list_entries) {
|
||||
free(g_view_list);
|
||||
g_view_list = NULL;
|
||||
}
|
||||
|
||||
#if EXPERIMENTAL_QUORUM_DEVICE_API
|
||||
if ((display_qdevice) && (v_handle)) {
|
||||
int err;
|
||||
struct votequorum_qdevice_info qinfo;
|
||||
|
||||
err = votequorum_qdevice_getinfo(v_handle, nodeid, &qinfo);
|
||||
if (err != CS_OK) {
|
||||
fprintf(stderr, "votequorum_qdevice_getinfo error: %d\n", err);
|
||||
} else {
|
||||
if (nodeid_format == NODEID_FORMAT_DECIMAL) {
|
||||
printf("%10u ", VOTEQUORUM_NODEID_QDEVICE);
|
||||
} else {
|
||||
printf("0x%08x ", VOTEQUORUM_NODEID_QDEVICE);
|
||||
}
|
||||
printf("%3d %s (%s)\n", qinfo.votes, qinfo.name, qinfo.state?"Voting":"Not voting");
|
||||
}
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
static int display_quorum_data(int is_quorate, int loop)
|
||||
static const char *decode_state(int state)
|
||||
{
|
||||
switch(state) {
|
||||
case NODESTATE_MEMBER:
|
||||
return "Member";
|
||||
break;
|
||||
case NODESTATE_DEAD:
|
||||
return "Dead";
|
||||
break;
|
||||
case NODESTATE_LEAVING:
|
||||
return "Leaving";
|
||||
break;
|
||||
default:
|
||||
return "Unknown state (this is bad)";
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
static int display_quorum_data(int is_quorate, uint32_t nodeid, int loop)
|
||||
{
|
||||
struct votequorum_info info;
|
||||
int err;
|
||||
@ -335,8 +377,9 @@ static int display_quorum_data(int is_quorate, int loop)
|
||||
return CS_OK;
|
||||
}
|
||||
|
||||
if ((err=votequorum_getinfo(v_handle, 0, &info)) == CS_OK) {
|
||||
if ((err=votequorum_getinfo(v_handle, nodeid, &info)) == CS_OK) {
|
||||
printf("Node votes: %d\n", info.node_votes);
|
||||
printf("Node state: %s\n", decode_state(info.state));
|
||||
printf("Expected votes: %d\n", info.node_expected_votes);
|
||||
printf("Highest expected: %d\n", info.highest_expected);
|
||||
printf("Total votes: %d\n", info.total_votes);
|
||||
@ -348,6 +391,12 @@ static int display_quorum_data(int is_quorate, int loop)
|
||||
if (info.flags & VOTEQUORUM_INFO_LAST_MAN_STANDING) printf("LastManStanding ");
|
||||
if (info.flags & VOTEQUORUM_INFO_AUTO_TIE_BREAKER) printf("AutoTieBreaker ");
|
||||
if (info.flags & VOTEQUORUM_INFO_LEAVE_REMOVE) printf("LeaveRemove ");
|
||||
if (info.flags & VOTEQUORUM_INFO_QDEVICE) {
|
||||
printf("Qdevice ");
|
||||
display_qdevice = 1;
|
||||
} else {
|
||||
display_qdevice = 0;
|
||||
}
|
||||
printf("\n");
|
||||
} else {
|
||||
fprintf(stderr, "votequorum_getinfo FAILED: %d\n", err);
|
||||
@ -361,7 +410,7 @@ static int display_quorum_data(int is_quorate, int loop)
|
||||
* 0 if not quorate
|
||||
* -1 on error
|
||||
*/
|
||||
static int show_status(nodeid_format_t nodeid_format, name_format_t name_format)
|
||||
static int show_status(uint32_t nodeid, nodeid_format_t nodeid_format, name_format_t name_format)
|
||||
{
|
||||
int is_quorate;
|
||||
int err;
|
||||
@ -395,11 +444,11 @@ quorum_err:
|
||||
return -1;
|
||||
}
|
||||
|
||||
err = display_quorum_data(is_quorate, 0);
|
||||
err = display_quorum_data(is_quorate, nodeid, 0);
|
||||
if (err != CS_OK) {
|
||||
return -1;
|
||||
}
|
||||
display_nodes_data(nodeid_format, name_format);
|
||||
display_nodes_data(nodeid, nodeid_format, name_format);
|
||||
|
||||
return is_quorate;
|
||||
}
|
||||
@ -410,7 +459,7 @@ static int monitor_status(nodeid_format_t nodeid_format, name_format_t name_form
|
||||
|
||||
if (q_type == QUORUM_FREE) {
|
||||
printf("\nQuorum is not configured - cannot monitor\n");
|
||||
return show_status(nodeid_format, name_format);
|
||||
return show_status(our_nodeid, nodeid_format, name_format);
|
||||
}
|
||||
|
||||
err=quorum_trackstart(q_handle, CS_TRACK_CHANGES);
|
||||
@ -429,8 +478,8 @@ static int monitor_status(nodeid_format_t nodeid_format, name_format_t name_form
|
||||
}
|
||||
time(&t);
|
||||
printf("date: %s", ctime((const time_t *)&t));
|
||||
err = display_quorum_data(g_quorate, loop);
|
||||
display_nodes_data(nodeid_format, name_format);
|
||||
err = display_quorum_data(g_quorate, our_nodeid, loop);
|
||||
display_nodes_data(our_nodeid, nodeid_format, name_format);
|
||||
printf("\n");
|
||||
loop = 1;
|
||||
if (err != CS_OK) {
|
||||
@ -463,13 +512,37 @@ static int show_nodes(nodeid_format_t nodeid_format, name_format_t name_format)
|
||||
}
|
||||
}
|
||||
|
||||
display_nodes_data(nodeid_format, name_format);
|
||||
display_nodes_data(our_nodeid, nodeid_format, name_format);
|
||||
|
||||
result = EXIT_SUCCESS;
|
||||
err_exit:
|
||||
return result;
|
||||
}
|
||||
|
||||
static int unregister_qdevice(void)
|
||||
{
|
||||
#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
|
||||
int err;
|
||||
struct votequorum_qdevice_info qinfo;
|
||||
|
||||
err = votequorum_qdevice_getinfo(v_handle, our_nodeid, &qinfo);
|
||||
if (err != CS_OK) {
|
||||
fprintf(stderr, "votequorum_qdevice_getinfo FAILED: %d\n", err);
|
||||
return -1;
|
||||
}
|
||||
|
||||
err = votequorum_qdevice_unregister(v_handle, qinfo.name);
|
||||
if (err != CS_OK) {
|
||||
fprintf(stderr, "votequorum_qdevice_unregister FAILED: %d\n", err);
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
#else
|
||||
fprintf(stderr, "votequorum quorum device support is not built-in\n");
|
||||
return -1;
|
||||
#endif
|
||||
}
|
||||
|
||||
/*
|
||||
* return -1 on error
|
||||
* 0 if OK
|
||||
@ -509,6 +582,11 @@ static int init_all(void) {
|
||||
goto out;
|
||||
}
|
||||
|
||||
if (cmap_get_uint32(cmap_handle, "runtime.votequorum.this_node_id", &our_nodeid) != CS_OK) {
|
||||
fprintf(stderr, "Unable to retrive this node nodeid\n");
|
||||
goto out;
|
||||
}
|
||||
|
||||
return 0;
|
||||
out:
|
||||
return -1;
|
||||
@ -530,12 +608,13 @@ static void close_all(void) {
|
||||
}
|
||||
|
||||
int main (int argc, char *argv[]) {
|
||||
const char *options = "VHslme:v:hin:d:";
|
||||
const char *options = "VHslmfe:v:hin:d:";
|
||||
char *endptr;
|
||||
int opt;
|
||||
int votes = 0;
|
||||
int ret = 0;
|
||||
uint32_t nodeid = VOTEQUORUM_NODEID_US;
|
||||
uint32_t nodeid = 0;
|
||||
uint32_t nodeid_set = 0;
|
||||
nodeid_format_t nodeid_format = NODEID_FORMAT_DECIMAL;
|
||||
name_format_t address_format = ADDRESS_FORMAT_NAME;
|
||||
command_t command_opt = CMD_UNKNOWN;
|
||||
@ -552,6 +631,14 @@ int main (int argc, char *argv[]) {
|
||||
|
||||
while ( (opt = getopt(argc, argv, options)) != -1 ) {
|
||||
switch (opt) {
|
||||
case 'f':
|
||||
if (using_votequorum() > 0) {
|
||||
command_opt = CMD_UNREGISTER_QDEVICE;
|
||||
} else {
|
||||
fprintf(stderr, "You cannot unregister quorum device, corosync is not using votequorum\n");
|
||||
exit(2);
|
||||
}
|
||||
break;
|
||||
case 's':
|
||||
command_opt = CMD_SHOWSTATUS;
|
||||
break;
|
||||
@ -582,15 +669,18 @@ int main (int argc, char *argv[]) {
|
||||
break;
|
||||
case 'n':
|
||||
nodeid = strtol(optarg, &endptr, 0);
|
||||
if ((nodeid == 0 && endptr == optarg) || nodeid <= 0) {
|
||||
if ((nodeid == 0 && endptr == optarg) || nodeid < 0) {
|
||||
fprintf(stderr, "The nodeid was not valid, try a positive number\n");
|
||||
exit(2);
|
||||
}
|
||||
nodeid_set = 1;
|
||||
break;
|
||||
case 'v':
|
||||
if (using_votequorum() > 0) {
|
||||
votes = strtol(optarg, &endptr, 0);
|
||||
if ((votes == 0 && endptr == optarg) || votes < 0) {
|
||||
fprintf(stderr, "New votes value was not valid, try a positive number or zero\n");
|
||||
exit(2);
|
||||
} else {
|
||||
command_opt = CMD_SETVOTES;
|
||||
}
|
||||
@ -600,10 +690,12 @@ int main (int argc, char *argv[]) {
|
||||
exit(2);
|
||||
}
|
||||
break;
|
||||
case ':':
|
||||
case 'h':
|
||||
case '?':
|
||||
default:
|
||||
break;
|
||||
command_opt = CMD_UNKNOWN;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
@ -616,9 +708,15 @@ int main (int argc, char *argv[]) {
|
||||
ret = show_nodes(nodeid_format, address_format);
|
||||
break;
|
||||
case CMD_SHOWSTATUS:
|
||||
ret = show_status(nodeid_format, address_format);
|
||||
if (!nodeid_set) {
|
||||
nodeid = our_nodeid;
|
||||
}
|
||||
ret = show_status(nodeid, nodeid_format, address_format);
|
||||
break;
|
||||
case CMD_SETVOTES:
|
||||
if (!nodeid_set) {
|
||||
nodeid = our_nodeid;
|
||||
}
|
||||
ret = set_votes(nodeid, votes);
|
||||
break;
|
||||
case CMD_SETEXPECTED:
|
||||
@ -627,6 +725,9 @@ int main (int argc, char *argv[]) {
|
||||
case CMD_MONITOR:
|
||||
ret = monitor_status(nodeid_format, address_format);
|
||||
break;
|
||||
case CMD_UNREGISTER_QDEVICE:
|
||||
ret = unregister_qdevice();
|
||||
break;
|
||||
}
|
||||
|
||||
close_all();
|
||||
|
||||
Loading…
Reference in New Issue
Block a user