In a 10-node cluster where all nodes are booting up and starting corosync
at the same time, sometimes during this process corosync detects a node as
leaving and rejoining the cluster.
Occasionally the downlist that gets picked contains the local node. When the
local node sends leave events for the downlist (including itself), it sets
its cpd state to CPD_STATE_UNJOINED and clears the cpd->group_name. This
means it no longer sends CPG events to the CPG client.
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
As a corosync-newbie it can be hard to bridge the gap between where a
particular message is sent and where the receive handler processes it,
and vice versa.
Reviewed-by: Angus Salkeld <asalkeld@redhat.com>
IPC: return 0/-ENOBUFS from message handler
IPC: use the new rate_limit API to improve perf.
CPG: add send_async API & hook up flow control
IPC: Fix flow control getting stuck.
IPC: Port the remaining libs to use libqb IPC
IPC: remove libqb flowcontrol API
TEST: put cpg_dispatch() in it's own thread
IPC: cleanup ipc_glue.c name everything cs_ipcs_*()
IPC: add back statistics
IPC: remove coroipcc_ symbols from lib*.versions
IPC: init each se's IPC as it is loaded.
IPC: use the new connection_closed() event to free the context.
IPC: re-add zero copy functionality back
IPC: remove cpg_mcast_joined_async() and make it the default
-> now cpg_mcast_joined() == cpg_mcast_joined_async()
libqb: expose a libqb error converter
libqb: add missing error conversions
libqb: remove repeat try loop in lib/cpg.c
CPG: fix zero copy mcast
CPG: use newer return codes
Add ENOTCONN to qb_to_cs_error()
libqb: fix error conversion from errno to cs_error_t in confdb
libqb: change errno_to_cs to qb_to_cs_error
libqb: add a cs_strerror() to get a more meaningful message
libqb: fix some confusing error conversions.
libqb: set the timeout on recv's to -1 (wait forever)
Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>
Following situation could happen:
- one thread is waiting for finish write operation (line 853), objdb is
locked
- flush (done in objdb_notify_dispatch) is called in main thread, but
this call will never appear because main thread is waiting for objdb
lock.
In this situation deadlock appears.
Commit solves this by:
- setting pipe to non-blocking mode
- pipe is used only as trigger for coropoll
- dispatch messages are stored in list
- main thread is processing messages from list
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>
in confdb_object_iter result of object_find_create is now properly
checked. object_find_create can return -1 if object doesn't exists.
Without this check, incorrect handle (memory garbage) was directly
passed to object_find_next.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Angus Salkeld <asalkeld@redhat.com>
In this concrete case result is equivalent but makes coverity happy.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>
If one node is paused it can miss a config change and
thus report a larger old_members than expected.
The solution is to use the left_nodes field.
Master selection used to be "choose node with":
1) largest previous membership
2) (then as a tie-breaker) node with smallest nodeid
New selection:
1) largest (previous #nodes - #nodes know to have left)
2) (then as a tie-breaker) node with smallest nodeid
Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Without refcounting the conn pointer here, corosync will segfault
if one kills a running instance of "corosync-cfgtool -r" (rhbz#695191)
Signed-off-by: Tim Serong <tserong@novell.com>
Reviewed-by: Steven Dake <sdake@redhat.com>
If you are connected to corosync and registered for
object notifications then corosync is asked to shutdown
the IPC server will get stuck. This is because the pipe
is closed and the refcount is increased. This leaves ipcs
with a connection that it can't destroy.
Solution:
1) if a write to the pipe fails (pipe closed) decrement the refcounter.
2) fix the object_track_stop() - it was not working as the functions
did not match up. (this caused the late callbacks).
3) in ipcs call exit_fn() then stats_destroy_connection() so that
the service engine can have time to call object_track_stop()
before the object gets destroyed.
Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>
Zero element array behavior is very different from normal array or
pointer. This behavior is root of problem in not returning correctly
filled array of addresses. This appeared only in rrp mode, where more
then one address is returned.
All memcpy's are now correctly converted to copy pointer to char.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>
corosync-notifyd has exposed an issue with confdb notifications.
The normal state of affairs is:
IPC thread > lock > objdb > lock
objdb notification whilst really useful turn things around:
<middle of big call chain>
objdb > lock > confdb > ipc > lock
This reverse ordering of locks causes a horrible dead lock.
I see this patch as a work around until corosync-2.0
when most of the threads and locking disappear.
This patch adds a pipe to confdb service. When we get a
objdb notification a struct gets written to the pipe.
The poll loop then runs the dispatch in the main thread.
In the dispatch we call the real ipc_dispatch_send().
Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>
- timestamps -> uint64_t and in nanosecs
- use clock_gettime
- common object naming
- common state names
- timeouts in milliseconds
git-svn-id: http://svn.fedorahosted.org/svn/corosync/trunk@3054 fd59a12c-fef9-0310-b244-a6a79926bd2f
Send CPG_REASON_PROCDOWN on process left
Our manual pages are clear:
CPG_REASON_PROCDOWN - the process left a group without calling
cpg_leave().
Currently, we are sending CPG_REASON_LEAVE in such situation.
git-svn-id: http://svn.fedorahosted.org/svn/corosync/trunk@2946 fd59a12c-fef9-0310-b244-a6a79926bd2f
Problem:
Under certain circumstances cpg does not send group leave messages.
With a big token timeout (tested with token == 5min).
1 start all nodes
2 start ./test/testcpg on all nodes
2 go to the node with the lowest nodeid
3 ifconfig <int> down && killall -9 corosync && /etc/init.d/corosync restart && ./testcpg
4 the other nodes will not get the cpg leave event
5 testcpg reports an extra cpg group (basically one was not removed)
Solution:
If a member gets removed using the new trans_list and
that member is the node used for syncing (lowest nodeid)
then the next lowest node needs to be chosen for syncing.
git-svn-id: http://svn.fedorahosted.org/svn/corosync/trunk@2785 fd59a12c-fef9-0310-b244-a6a79926bd2f
Patch adds new function to initialize cpg, cpg_model_initialize. Model
is set of callbacks. With this function, future addions of models
should be possible without changing the ABI.
Patch also contains callback in CPG_MODEL_V1 for notification about
Totem membership changes.
git-svn-id: http://svn.fedorahosted.org/svn/corosync/trunk@2770 fd59a12c-fef9-0310-b244-a6a79926bd2f
Add support for MESSAGE_REQ_CPG_FINALIZE message. This will allow us
remove cpg_pd from list of active connections, and remove problem, when
cpg_finalize + cpg_initialize + cpg_join can result in CPG_ERR_EXIST
error.
git-svn-id: http://svn.fedorahosted.org/svn/corosync/trunk@2676 fd59a12c-fef9-0310-b244-a6a79926bd2f
Patch handles situation, when on one node, one process:
- join cpg
- do same actions
- leave cpg
- join cpg again
Following sequence can (racy) end with broken process_info list.
To solve this problem, one more check is done in
message_handler_req_lib_cpg_join so if process_info with same pid and
group as new join request exists, CPG_ERR_TRY_AGAIN is returned.
git-svn-id: http://svn.fedorahosted.org/svn/corosync/trunk@2675 fd59a12c-fef9-0310-b244-a6a79926bd2f
calling cfg_track_stop. This caused corosync to crash.
The extra list_empty() check is redundant too because it also happens in remove_ci_from_shutdown()
git-svn-id: http://svn.fedorahosted.org/svn/corosync/trunk@2655 fd59a12c-fef9-0310-b244-a6a79926bd2f
- if a single node is booted with votequorum loaded then
corosync-quorumtool shows zero nodes and no votes.
- votequorum doesn't always tell the main quorum module when a new node
has joined the cluster (principally itself. this bug is actually tied
into the above)
I've also added quorum to the default list of services. As quorum has
been decoupled from sync it will not interfere with normal operations as
it used to do and it makes more sense to have it there than not.
git-svn-id: http://svn.fedorahosted.org/svn/corosync/trunk@2510 fd59a12c-fef9-0310-b244-a6a79926bd2f
