Token rx and tx timestamps were computed and stored as 32-bit unsigned
integer but substracted in other parts of code from 64-bit integer.
Result was, that node with uptime larger than 49.71 days
(2^32/(1000*60*60*24)) reported wrong numbers for
stats.srp.time_since_token_last_received and in log message during long
pause (function timer_function_orf_token_warning).
Solution is to store rx and tx data as 64-bit integer.
Fixes#761
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
totem.knet_mtu is new configuration option which allows setting
of automatic or manual knet MTU.
Also reload of totem.knet_pmtud_interval is fixed now, so it works when
key is deleted (and set back default value).
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Before this, all knet messages, including debug, were sent
over the pipe from knet to corosync and filtered in corosync.
This was obviously a waste, so now we tell knet the logging
level we need from it and so only get the messages that the
user has requested.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Previously, existence of retransmit messages canceled holding
of token (and never allowed representative to enter token hold
state).
This makes token rotating maximum speed and keeps processor
resending messages over and over again - overloading network
and reducing chance to successfully deliver the messages.
Also there were reports of various Antivirus / IPS / IDS which slows
down delivery of packets with certain sizes (packets bigger than token)
what make Corosync retransmit messages over and over again.
Proposed solution is to allow representative to enter token hold
state when there are only retransmit messages. This allows network to
handle overload and/or gives Antivirus/IPS/IDS enough time scan and
deliver packets without corosync entering "FAILED TO RECEIVE" state and
adding more load to network.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
CFG tracking was removed in 815375411e,
probably as a mistake, as part of the tidy up of cfg and the removal of
dynamic loading. This means that shutdown tracking (using
cfg_try_shutdown()) stopped working.
This patch restores the trackstart & trackstop API calls (renamed to be
more consistent with the exiting libraries) so that shutdown tracking
can be used again.
Change cfg.shutdown_timeout to be in milliseconds rather than seconds
nd use libqb macros for conversion.
Add --force option to corosync-cfgtool -H
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Patch tries to make nodestatusget really extendable. Following changes
are implemented:
- corosync_cfg_node_status_version_t is added with (for now) single
value CFG_NODE_STATUS_V1
- corosync_knet_node_status renamed to corosync_cfg_node_status_v1 (it
isn't really knet because it works as well for udp(u()
- struct res_lib_cfg_nodestatusget_version is added which holds only ipc
result header and version on same position as for
corosync_cfg_node_status_v1
- corosync_cfg_node_status_get requires version and pointer to one of
corosync_cfg_node_status_v structures
- request is handled in case switches to make adding new version easier
Also fix following bugs:
- totempg_nodestatus_get error was retyped to cs_error_t without any
meaning.
- header.error was not checked at all in the library
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Current we horribly over-use totempg_ifaces_get() to
retrieve information about knet interfaces. This is an attempt to
improve on that.
All transports are supported (so not only Knet but also UDP(U)).
This patch builds best against the "onwire-upgrade" branch of knet
as that's what sparked my interest in getting more information out.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Current quorum callback contains only actual view list and there is no
way how to find out joined/left nodes. This cannot be emulated by user
app, because when corosync restarts before other nodes notices then view
list is unchanged (ring id is changed tho).
Solution is to implement similar callback as for cpg which contains ring
id, member list, joined list and left list.
To implement such callback and keep backwards compatibility,
quorum_model_initialize is introduced. Its behavior is similar to
cpg_model_initialize. This allows passing model v1, which contains
enhanced quorum (full ring id is passed instead of just seq number)
and nodelist callbacks.
To find out which events should be sent by corosync daemon, new message
MESSAGE_REQ_QUORUM_MODEL_GETTYPE is used. Quorum library on init was
sending MESSAGE_REQ_QUORUM_GETTYPE. Whem model v1 is requested the
MESSAGE_REQ_QUORUM_MODEL_GETTYPE is used, which contains model number
so corosync knows that client is using model v1 and can send enhanced
quorum and nodelist events.
Nodelist event is (for now) send both in case of change of membership
and also when requested, also when CS_TRACK_CURRENT is requested, but
then left_list and joined_list is left empty, because they don't make
too much sense there.
New test application testquorummodel is added as an example of new API
usage.
Also during patch developement, I found few bugs here and there, which
are also fixed:
- quorum_initialize was never returning error code returned by
MESSAGE_REQ_QUORUM_GETTYPE call (always returned CS_OK)
- Allocated memory in send_library_notification was based
on sizeof(unsigned int) instead of mar_uint32_t. That's not wrong,
but it make more sense to use sizeof(mar_uint32_t) instead
(big thanks to Chrissie for englishify the man pages)
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Needs new knet crypto API.
If it's not available, then fall back to the old
API and forbid changing crypto while running.
To avoid us being dependant on the leader node, each
node sends its own crypto_reconfig_phase messages so
we can guarantee that the reconfiguration always completes
on each node.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Have string values stored in char arrays in totem_config
so we don't get into a mess with the pointers.
Also remove vsftype (which hasn't been used since corosync 1)
Use strncpy even though we know the string is fine. Keep covscan happy
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
To be more reliable & maintainable
The basic plan here is to fix reloads to be more stable
using read/parse/verify/build/commit stages, so that any errors
will not leave corosync in an unstable state. This should
also make the code more maintainable as currently the verify/commit
stages are horribly intertwined.
Also:
- Fix local_node_pos not being updated in the new map during validation
(broke adding and removing new nodes in the middle of the list).
- Fix reconfiguration so that nodes are indexed by nodeid and not their
position in the list. This is an old bug that's just been carried
over
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
This reverts commit 934c47ed43 which is
causing protocol incompatibility in needle. Master seems to be not
affected, but it needs more checking.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
sin6_scope_id was not present in totemip structure making impossible to
use link-local ipv6 address.
Patch adds sin6_scope_id and changes convert/copy functions to use it
(formally also comparator functions should be changed, but it seems to
cause more harm and it is not really needed).
This makes corosync work with link-local addresses fine for both UDPU
and Knet transport as long as interface specification is used (so
fe80::xxxx:xxxx:xxxx:xxxx%eth0).
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Make the icmap*_r functions read from the specified map rather
than the global map.
Also include icmap_get_string_r() which seems to have been missed out.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Previously node id was logged ether as a %d (most often), %u, %x or
PRI.32 and ring id ether as %lld, %llx with various separators (., :, /)
between rep nodeid and seq. This seems to cause confusion.
This patch adds macros CS_PRI_NODE_ID, CS_PRI_RING_ID and
CS_PRI_RING_ID_SEQ (CS prefix = corosync, PRI modeled in spirit of
inttypes.h PRIx32) and makes code use them.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
This feature allows corosync to block packets received from unknown
nodes (nodes with IP address which is not in the nodelist). This is
mainly for situations when "forgotten" node is booted and tries to join
cluster which already removed such node from configuration. Another use
case is to allow atomic reconfiguration and rejoin of two separate
clusters.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Previously the reason field for the member_list items
in cpg_totem_confchg_fn was unset what may be little confusing.
Solution is to add a special value CPG_REASON_UNDEFINED and use it for
the member_list items.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
AF_UNSPEC returns different results than AF_INET/AF_INET6, because of
nsswitch.conf search is in order and it stops asking other
modules once current module success.
Example of difference between previous and new code when ipv6-4 is used:
- /etc/hosts contains test_name with an ipv4
- previous code called AF_INET6 where /etc/hosts failed so other methods
were used which may return IPv6 addr -> result was ether fail or IPv6
address.
- new code calls AF_UNSPEC returning IPv4 defined in /etc/hosts ->
result is IPv4 address
New code behavior should solve problems caused by nss-myhostname.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Originally totem.ip_version was used to force ip version used by totem.
With Knet this variable didn't make too much sense so it was not used.
Sadly rely only on DNS resolver order doesn't always work (RFC is quite
complicated, but if IPv6 is not configured then IPv4 is preferred), what
we tried to solve by forcing IPv6 and only if that fails, use IPv4.
Sadly this collides with nss_myhostname which is able to return every
local address and today system usually have at least one autogenerated
link-local IPv6 address so it is able to "overwrite" /etc/hosts.
Solution is to enhance totem.ip_version and use it also for Knet.
totem.ip_version is now just a flag for resolver and can have four
states: ipv4 (only IPv4 is used), ipv6 (only IPv6 is used), ipv4-6 (ask
IPv4 first and if it fails ask for IPv6) and ipv6-4 (ask IPv6 first and
if it fails ask for IPv4). Default for Knet and UDPU transports is
ipv6-4, for UDP it's ipv4, because autogenerated mcast addr doesn't play
too well with ipv6-4.
So everywhere where nss_myhostname becomes problem, it's just possible
to set totem.ip_version to ipv4-6.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Feature depends on existence of libqb function qb_log_file_reopen.
New function call is added into CFG service API. This function is
used by corosync-cfgtool which now accepts -L parameter.
Finally, logrotate "postrotate" script is calling
corosync-cfgtool -L to notify corosync, instead of using
copytruncate option.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Token_warning is used to present information about
when the token was last received.
Signed-off-by: Chris Walker <cwalker@cray.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
cmap_iter_next in contrast of it's icmap counterpart copies key name
into user preallocated space. In the worst case, key name may be
CMAP_KEYNAME_MAXLEN, so cmap_iter_next then need CMAP_KEYNAME_MAXLEN +
additional byte to store zero. strncpy was copying only
CMAP_KEYNAME_MAXLEN characters so there was possibility of unterminated
string.
Patch solves this by using memcpy and always add trailing zero.
Documentation was improved suggesting minimum size of keyname buffer to
be CMAP_KEYNAME_MAXLEN + 1.
Also sam and quorumtool were using too short buffer so they are fixed too.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
To make finding victim of incompatible messages easier, IP of sender is
logged. Propagating IP in layers makes patch slightly larger.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Magic number (0xC070) together with version in every packet
is used for detecting that other node is really
Corosync 3.x.
Endian_detector field is removed and magic number is now
used instead.
If received packet magic number differs, guessing is used to show more
about the source (Corosync 2.3+, 2.2 are quite reliable, Knet and
unencrypted Corosync 2.1/2.0/1.x/OpenAIS are semi-reliable and encrypted
Corosync 2.1/2.0/1.x/OpenAIS are quite unreliable).
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Even if it's not used for anything else.
Also, make cfgtool show the correct link ID when links are not
contiguous
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
This shrinks the srp_addr (and consequently every packet sent by
corosync) so that instead of containing loads of IP addresses to
identify a node, it just sends the nodeid.
This then allows us to make ring0 optional and replaceable when running
knet.
It also means that we need some other way of identifying the local
node in corosync.conf, so the nodelist.node.name entry is now mandatory
and is mapped to the local host using the same algorithm as used in
cman.
This code needs LOTS of testing as it touches a huge amount of totemsrp
and totemconfig.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
KNET has options for nss or openssl crpyto libraries, make this
available to corosync.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Only add the knet handle stat keys if we are actually running knet. This
prevents errors occurring when iterating through all of the stats keys
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
nodeid in struct req_lib_cfg_get_node_addrs is "unsigned int",
so the function corosync_cfg_get_node_addrs should have its param
"nodeid" to be unsigned int.
Signed-off-by: Bin Liu <bliu@suse.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Previously, corosync executable was repeatedly (proportionally to the
count of LOGSYS_DECLARE_SUBSYS macro applications involved in the
constituent source files) checking the same for no gain in the pre-main
startup. This is not needed since nothing changes with static data
shared withing the same program space (it may have been a different
story once upon a time if loadable modules were in use), so make that
happen in (one-off per executable) LOGSYS_DECLARE_SYSTEM instead.
Libqb offers it's own ready-made macro to that
effect, simply to isolate the inner percularities from the library user
(that should not be required to understand anything about the orphan
sections and respective autocreated symbols to denote their boundaries).
As it is compile-time conditionalized in the same way, just use it
directly instead. As a value added, corosync will be kept up to date
about the possibly growing set of the logging-sanity checks as it gets
compiled with newer and newer libqb versions (their header files, for
that matter).
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Now we are using knet, it's possible to dynamically add, remove and
reconfigure links on the fly.
Also print 'n' for non-existant knet links. This will show up
only on loopback links >0. But it looks better than 'status ='
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
If bind call fails it's retried for BIND_MAX_RETRIES.
If it's still unsuccessful, corosync exists instead
of working incorrectly.
Slightly modified by reviewer.
Signed-off-by: Masse Nicolas <nicolas.masse@stormshield.eu>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
knet handle stats show compression and crypto statistics. With these
you can see the effectiveness of compression and the overheads of both
crypto and compression.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
RRP doesn't exist any more so all the ring re-enable code is redundant.
I've removed it from the library and all the code that does anything,
but I've left the hole in the IPC just in case old libraries are
hanging around.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Icmap is factored out so it's possible to add other
maps for cmap. API call to switch maps from application
end is added.
Corosync-cmapctl is enhanced with -m option.
Stats contains all statistics previously found in runtime.connections,
runtime.services and runtime.totem prefixes together with new knet
related. All stats are read only.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
Knet needs longer key and supports various key lengths. Split
TOTEM_PRIVATE_KEY_LEN into TOTEM_PRIVATE_KEY_LEN_MIN and
TOTEM_PRIVATE_KEY_LEN_MAX (both using KNET_*_KEY_LEN).
Fix incorrect "Could only read..." message.
Make sure key is properly initialized/zeroed.
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Unsigned int and size_t represent two different concepts.
Same problem was present in ipc_glue.
Signed-off-by: Michael Jones <jonesmz@jonesmz.com>
Reviewed-by: Jan Friesse <jfriesse@redhat.com>
knet needs buffers to be KNET_MAX_PACKET_SIZE or messages will
get lost or corrupted.
UDPU packets shouldn't be that big so I introduced UDP_FRAME_SIZE_MAX
for that transport.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
Corosync layers don't need to know the knet MTU size - this way
corosync fragments buffers only when they get larger than the
KNET buffer size (64K) and knet fragments below that based on
the actual MTU and transport considerations.
It is also now possible to configure knet to use UDP or SCTP
transports in corosync.conf. This is currently done per-link
so if you have more than 1 link you need several interface{}
stanzas inside totem{} to make it use other than the default
of UDP. if it's useful I might add the option of a global
default.
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>