Add --disable-tests to allow the user to disable tests. As a
side-effect, this will avoid the following build failure when check is
found:
libstat_wrapper.c:11:10: fatal error: gnu/lib-names.h: No such file or directory
11 | #include <gnu/lib-names.h>
| ^~~~~~~~~~~~~~~~~
This build failure is raised since version 2.0.5 and
78df90b180
Fixes:
- http://autobuild.buildroot.org/results/450cfc36d4fd6dc71c138bec45f05b5a2d92a08d
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
... to make 2.71 happy. Also increase minimum version to 2.69 (10 years
old version so should be compatible enough).
Signed-off-by: Jan Friesse <jfriesse@redhat.com>
* doxygen2man: Add option to read copyright line from the header file
This should help make builds reproducible.
I tried various methods of getting the date, using 'git' is no use as
it could be run from a tarball, using the file date doesn't work either
so this seems a reasonable compromise.
* man: Use SOURCE_EPOCH to make manpage dates reproducible
Also add build-aux/update-copyright.sh to keep header file
copyright lines up-to-date.
All code taken from knet
Using of posix_fallocate() guarantees that, if it succeed, the
attempting to write to allocated space range does not fail because of
lack of storage space. This prevents SIGBUS when trying to write to
mmaped file and no space left.
Co-Authored-by: Ivan Zakharyaschev <imz@altlinux.org>
Reported-by: Mikhail Kulagin <m.kulagin at postgrespro dot ru>
Co-authored-by: Ivan Zakharyaschev <imz@altlinux.org>
* man: Use doxygen2man to make the API manpages
* [build] drop unused target
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
* [build] update gitignore
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
* [docs] make build -j safe and drop unnecessary dox config files
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
* man: Use doxygen2man to generate *.h.3 man pages
so we don't need to use the awful doxygen ones
Co-authored-by: Fabio M. Di Nitto <fdinitto@redhat.com>
* doxygen2man: Add utility to generate man pages from doxygen
This is in here from kronosnet so it cna be used by other parts
of the cluster stack.
* [man] drop trailing white spaces
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
* [build] cleanup variable names
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
* [build] add conditionals to use internal or external doxygen2man
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
* Update .gitignore
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Co-authored-by: Christine Caulfield <ccaulfie@redhat.com>
Now that splint is actually contradicting errors that come from
the compilers I think it's time to retire it. I could cope with it
being a minor nuisance on the argument that "another check can't
hurt", but contradicting the actual compilers is too much.
The CI has Coverity installed which is much more up-to-date anyway.
Splint hasn't been updated since 2010
this option is stricly meant for runtime debugging purposes.
do NOT use in production.
check gcc/clang man pages on how to use ASAN/UBSAN/TSAN.
Also allow to users to specificy SANITIZERS_CFLAGS and SANITIZERS_LDFLAGS
for advanced use.
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
There were a few missed leftovers in d6875f2 regarding compatibility
with sed on FreeBSD (some commands do require a newline and/or
backslash separation).
Merges: #335
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
move from AC_PREPROC_IFELSE (strongly discouraged) to AC_COMPILE_IFELSE
our detection system was very weak and recent versions of clang did
show that PREPROC_IFELFE (cpp) would enable warning options that
the compiler does not support (clang).
use a full compilation test to detect what works and what doesn't.
Also expand the warning list to include new / renamed clang options
of equivalents already enabled for older versions of clang and gcc.
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
* log: add systemd journal as a logging option
systemd journal can be configured as a logging option
at ./configure time (--enable-systemd-journal).
If libqb is buit with this then the syslog target can be switched
to sending to the journal using
qb_log_ctl(QB_LOG_SYSLOG, QB_LOG_CONF_USE_JOURNAL, 1);
It is my (and several others') opinion that the linker 'magic' used to maintain callsite data in libqb is hugely over complicated and unnecessarily fragile. It's main purpose seems to have been to improve performance but empirical testing shows this to be tiny at best. The overhead of sprintf makes minor optimisations in this code pointless.
With this code removed, libqb allocates callsites using a C static variable at run-time. This sounds bad but in actuality it merely moves the allocation from program load time to the first few milliseconds of program run-time. Applications like corosync and pacemaker spend most of their time in small loops doing the same work over and over again so the overhead doesn't apply and jitter does not occur.
We've tested this with corosync and pacemaker under valgrind and massif and the differences are minimal and even then only show up under artificial stress testing.
For this change I've bumped the soname up to 20 to indicate this is an incompatible change. I'm open to suggestions as to a release number but am currently thinking of 2.0.0
* doc: qbarray.h: fix garbled Doxygen markup
* build: follow-up for and fine-tuning of a rushed 6d62b64 commit
(It made a service as-was, but being afforded more time, this would
have accompanied that commit right away, for better understanding,
brevity and uniformity.)
* build: prune superfluous Makefile declarations within tests directory
There was a significant redundancy wrt. build flags and EXTRA_DIST
assignment (the latter become redundant as of f6e4042 at latest)
spread all over the place (vivat copy&paste). Also, in one instance,
CPPFLAGS (used) was confused with CFLAGS (meant).
* maint: check abi: fix two issues with abi-compliance-checker/libstdc++
1. ABICC >= 2 needs to be passed -cxx-incompatible switch because C is
no longer a default for this tool (used to be vice versa),
plus current version will stop choking on C vs. C++ (our C code with
C++ compatibility wrapping being viewed from C++ perspective for the
purpose of dumping the declared symbols, which somewhat conflicts
with internal masking of the C++ keywords being used as valid C
identifiers [yet some instances must not be masked here, see
https://github.com/lvc/abi-compliance-checker/issues/64) only
if _also_ something like this is applied:
https://github.com/lvc/abi-compliance-checker/pull/70
2. since 20246f5, libqb.so no longer poses a symlink to the actual
version-qualified shared library, but rather a standalone linker
script, which confuses ABICC, so blacklist that file for the scanning
purposes explicitly, together with referring to the library through
it's basic version qualification (which alone, sadly, is not
sufficient as ABICC proceeds to scan whole containing directory
despite particular file is specified)
* maint: check abi: switch to abi-dumper for creating "ABI dumps"
Beside avoiding issues with abi-compliance-checker in the role of ABI
dumps producer (see the preceding commit), it also seems to generate
more accurate picture (maybe because it expressly requires compiling
with debugging information requested).
* Low: qblist.h: fix incompatibility with C++ & check it regularly
* tests: check_list.c: start zeroing in on the gaps in tests' coverage
* tests: print_ver: make preprocessor emit "note" rather than warning
IIRC, Chrissie asked about this around inclusion of the test at
hand, and it seemed there was no way but to emit a warning to get
something output at all. Now it turns wrong, and moreover, we
make the code not fixed on GCC specific pragmas, with a bit of
luck, "#pragma message" approach is adopted more widely by compilers.
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
* Replace ck_assert_uint_eq() with ck_assert_int_eq()
it's not available in check 0.9
* Proper check for C++ compiler (from Fabio)
* add (c) to copyright dates
This is meant as a lean, customized policy driven alternative
to the original proposal by Jan Friesse <jfriesse@redhat.com> that
balances the inherent trade-off in the opposite direction, so that the
configure.ac script is practically untouched and more weight and policy
is hardcoded in git-version-gen. Problem with that approach stems from
(avoidable) effective fork of the respective gnulib's module and imposed
maintenance burden.
Speaking for libqb in particular, we should nonetheless make it
absolutely clear such in-development snapshots are nothing more,
nothing binding (not to think of viral injection of these bits
into circle of dependent packages upon their rebuilds), since the
changes accumulated since the last official release should only be
assumed firmly committed at the point the new release is cut (may
happen 99% of time with snapshots but no accountability from our
side for the complementary inter-release twists...), which is exactly
when many possibly unanticipated variables like correct SONAME
versions get to reflect what's appropriate.
Also, OpenPGP signature constitutes something more eligible for
one's trust than (provably) bit/content unstable archives without
the possibility of an independent authenticity/integrity verification.
Therefore, the only thinkable and upstream-endorsed use cases
for such snapshots are development-only purposes (CI et al.)!
V2 of the patch:
Thanks to feedback from Jan Friesse, a glitch in "make rpm" et al.
not working with the snapshots was pointed out.
V3:
Only normalize configure.ac back when known to be previously affected
with "git archive" substitution logic, and do not use an exclamation
mark as short commit - decoration separator since that could be
ambiguous (it is a valid branch name character), stick with double
question marks instead (not allowed, doubled for good measure).
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
Uses dpkg-architecture, if present, to return
DEB_HOST_GNU_TYPE, and use this appended to /usr/include
for form the path.
Signed-off-by: Daniel Black <daniel@linux.ibm.com>
Reviewed-by: Jan Pokorný <jpokorny@redhat.com>
some vendors default to RPATH, others to RUNPATH. The former does not allow override
with LD_LIBRARY_PATH when running binaries such as test suite and it was causing
problems on platforms where RPATH is default, by running the test suite against
the wrong library (out-of-tree vs in-tree).
This change has no effect on libqb itself, but only on the binaries.
Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
For the former, a prototype and the final code got (hm, mysteriously)
intertwisted. For the latter, I am clearly guilty of (rare, anyway)
testing of the out-of-tree builds only with libqb-already-system-wide
scenario, which is rather shortsighted.
Thanks Fabio and his ci.kronosnet.org project for spotting that.
X-mas-present-for: Fabio M. Di Nitto <fdinitto@redhat.com>
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
(or rather [read on]: "bare" fix, now that we established means to
analyse the impact of the linker-dependent misbehaviour and to detect
some of its symptoms in preceding two commits, respectively)
Initially with the help of the internal test suite and the failing log
test, it was eventually discovered[1] that these binutils commits going
to the recent 2.29 release affected the treatment of _start_SECNAME
and __stop_SECNAME symbols denoting the boundary start/stop addresses
of a SECNAME orphan section -- specifically in libqb context a custom
section (SECNAME=__verbose) used for link-time ("run-time amortizing")
callsite collection when there's a support in the toolchain[*]:
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=cbd0eecf261c2447781f8c89b0d955ee66fae7e9https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=b27685f2016c510d03ac9a64f7b04ce8efcf95c4https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=7dba9362c172f1073487536eb137feb2da30b0ff
The first one explicitly states:
> Also __start_SECNAME and __stop_SECNAME symbols are marked as hidden
> by ELF linker so that __start_SECNAME and __stop_SECNAME symbols for
> section SECNAME in different modules are unique.
The problem is that libqb silently depends on the previous status quo
ld.bfd linker behaviour of keeping those symbols externally visible,
which was apparently not granted as it has deliberately changed per
above.
And then for 2.29.1 release of binutils once again, as someone actually
noticed something went overboard with the 2.29 changes:
http://lists.gnu.org/archive/html/bug-binutils/2017-08/msg00195.html
(overview of the original bug discussion, rather than directly
https://sourceware.org/bugzilla/show_bug.cgi?id=21964, which is
a result of a conflct resolution when restoring bugzilla backup)
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=487b6440dad57440939fab7afdd84a218b612796
At least that change doesn't invalidate all the effort being put into
the original version of the changeset, only the configure script check
had to be refined so as not to miss the "orphan section magic not
working properly out of the box, without band aid" observation
(see the inline comment) -- the workaround arrangement needs
to be applied in that case as well.
* * *
So regarding the solution itself, the core of the fix was sketched at
the original Fedora targeted bug against binutils[2]. In short, we are
using a custom linker script that (re)describes the mentioned custom
orphan output section, or better yet, assuredly pushes that section, and
more importantly, it's own boundary denoting symbols, through into the
resulting executable when it's being linked (as in compile-time step).
This solution alone, while working for the non-libqb (more on that
below) logging participants, is not good enough, as it requires all
libqb targets to start using new incantation (namely "-Wl,foo.t" switch)
in the final link step during compilation, which might be solvable
with a tweak in libqb's pkg-config file under assumption that practice
of using "pkg-config --libs libqb" is rigidly followed. Which is likely
a false expectation, and furthermore only for the regular consumption
model, as it doesn't cover the least bit the developmental one (refer
to previous-but-one "tests" commit message), e.g. applied for internal
examples + tests (but no local sub-checkout tree usage can be excluded).
So further extensions were devised to cover both consumption models:
- a. regular:
courtesy of binutils maintainer[3], we follow an idea to make libqb.so
(i.e. what the targets link against) rather a linker script on its
own, which first include the version-specified (e.g. libqb.so.0) file
into the link, then lists, in situ, the content of the linker script
per above, hence -lqb linking has the same effect as having both
"-lqb -Wl,foo.t" explicitly in the link command prior to this trick
- b. developmental:
to eliminate any kind of race condition arising from the attempt
to post-modify libqb.la libtool archive file generated internally
by libtool, we sort of abuse "inherited_linker_flags" variable
within this file format, as it forms an accumulative value across
the whole transitive dependencies chain (if not impaired per the
note below), fitting exactly our purpose of injecting "-Wl,foo.t"
switch equivalent for those libtool-linking by L{D,IB}ADD'ing
libqb.la; it's then enough to craft a custom libtool archive file
declaring that value, and hook it into such dependency chain through
libqb_la_LIBADD, and with a little bit of further fiddling, it works
as desired (note that double occurrence of "-Wl,foo.t" equivalent
present at some stages of sorting this trick turned out to be,
surprisingly, counter-productive, which should now demistify the
very existence of effectively empty qblog_script_noop.ld file);
NOTE: some forms of libtool distribution (debian + derivatives ones
in particular) undermine natural transitive dependency propagation
with a deliberate cut off (https://bugs.debian.org/702737), so we
need to ensure the "impairment" is not happening by force (corosync
precedent: https://github.com/corosync/corosync/commit/0f1dc5c1)
^ something like this needs to be applied for any such "private
consumer" (although it hopefully goes without saying this way
of consuming libqb outside of it's own playground is hardly
the Right Thing) if portability is important, nonetheless!
* * *
On the address of linker script workaround, there are linkers out there
that do not support the trick, for instance:
- ld.gold from binutils (but it has hardly ever been working with
orphan sections, anyway:
https://sourceware.org/bugzilla/show_bug.cgi?id=22291)
- ancient versions of ld.bfd, e.g. 10+ years old one used as a native
system linker even in the most recent releases of FreeBSD, unless
GCC toolchain is used instead
If these are hit when (because) the compiler has already demonstrated it
supports "section" attribute, the build system configuration is forcibly
stopped, simply to stay conceptually compatible with the prior state in
which the affinity to leverage that feature hasn't been called off
under any circumstances. One is, however, able to achieve exactly
this behaviour with --enable-nosection-fallback switch, but if some
other participants in the logging, possibly linked with a more friendly
linker, do utilize this orphan section, logging may silently break
(another reason to require an explicit sign-off).
Another note, the particular self-check change slightly touched in the
previous commit but otherwise predating this whole effort by far needs
to be modified now once again, this time because linker-script-based
workaround for newer linkers as stated causes the section boundary
symbols to be present regardless if that section is utilized, leading
to a self-inflicted breakage due to these empty section symbols suddenly
winning in the symbol resolution mechanism (previously the empty section
would be dropped incl. the boundary symbols), causing problems down the
line. It also makes this very check self-contained in the same
compilation unit that trigggers it, whereas previously it used to be the
said "arbitrary" winner and things kept silently working just because
failure condition -- empty section -- would be implicitly isolated.
Last but not least, libqb itself needs to be linked with the mentioned
"-Wl,foo.t" equivalent for its own outgoing log messages to be honoured
under all circumstances, which is already achieved with the arrangement
for b. above, and by experiments, further redefinition of those boundary
denoting symbols as weak was necessary so as to make them truly global
within libqb.so proper (at least with binutils 2.29).
* * *
To provide a high-level prioritized overview of what drove the approach:
- PRESERVATION OF BINARY COMPATIBILITY (ABI), which is achieved except
for a single "ABI nongracefulness" I am aware of but that's more
a consequence of slightly incorrect assumptions in the logic of
QB_LOG_INIT_DATA macro function predating this whole affair by
a long shot and which the patchset finally rectifies:
if in the run-time dynamic link, following is combined:
(. libqb, arbitrary variant: pre-/post-fix, binutils < / >= 2.29)
. an "intermediate" library (something that the end executable links
with) triggering QB_LOG_INIT_DATA macro and being built with
pre-fix libqb (and perhaps only with binutils < 2.29)
. end executable using no libqb's logging at all, but being built
with post-fix libqb (and arbitrary binutils < / >= 2.29)
then, unlike when executable is built with pre-fix libqb, the
special callsite data containing section in the ELF structure
of the executable is created + its boundary denoting symbols
defined within, despite the section being empty (did not happen
with pre-fix libqb), and because the symbols defined within the
target program have priority over that of shared libraries in the
symbol resolution fallback scheme, the assertion of QB_LOG_INIT_DATA
of the mentioned intermediate library will actually be evaluating
the inequality of boundaries for the section of the executable(!)
rather than it's own (or whatever higher prio symbols are hit,
presumably only present if the section at that level is non-empty,
basically a generalization of the story so far);
the problem then manifests as unability to run said executable
as it will fail because of the intermediate library inflicted
assertion (sadly with very unhelpful "Assertion `0' failed"
message);
fortunately, there's enough flexibility so as how to fix
this, either should be fine:
. have everything in the executable's library dependency closure
that links against libqb assurably (compile-time) linked with one
variant of libqb only (either all pre-fix or post-fix, mind the
apparent limitation of binutils' versions with the former)
. have the end executable (that does not use logging at all as
discussed precondition) linked using substitution like this:
s/-lqb/-l:libqb.so.0/ (you may need to adapt the number later)
and you may also need to add this CPPFLAG for the executable:
-DQB_KILL_ATTRIBUTE_SECTION
- as high level of isolation of the client space from the linker
(respectively toolchain) subtleties as possible (no new compilation
flags and such required, plus there's no way to hook any dynamic
computational ad-hoc decision when the compilation is about to
happen, anyway), and in turn, versatility is preserved as much as
possible
* * *
Finally, let's have a look how the already well-known test matrix
overview changes as of this commit, but first as a recap,
"X(Y)" denotes "X linked with linker Y":
X(a) .. ld.bfd < 2.29
X(b) .. ld.bfd = 2.29 (+ 2.29.1 and hopefully on)
and here you are (values in <angle brackets> denote non-trivial change
[not mere rewording] introduced as of this commit, in comparison to the
table stated in the preceding commit):
+=========+=========+=========+=========+=========+=========+=========+
#client(x)# libqb(a) usage # libqb(b) usage #
# vvv #---------+---------+---------+---------+---------+---------+
# V # direct | libX(a) : libX(b) # direct | libX(a) : libX(b) #
+=========+=========+=========+=========+=========+=========+=========+
# x = a # OK | OK : <OK> # <OK> | <OK> : <OK> #
# x = b # <OK> | <OK> : <OK> # <OK> | <OK> : <OK> #
+=========+=========+=========+=========+=========+=========+=========+
Everything is green \o/
* * *
Note: as of this fix, it is assumed that the non-green counterpart of
this table in the message for the preceding commit (loosely though[!],
as the occurrence of empty callsite section can no longer be attributed
to something bad going on as of this fix that enforces its presence
unconditionally, whereas it would be suppressed when unused before
with kind linkers, hence some other conditions can be witnessed
especially when QB_LOG_INIT_DATA misused in no-logging context)
doubles as an indicator how will mixing the logging participants wrt.
linker+libqb version work out, when "X(Y)" becomes read as "X linked
with linker Y under additional restriction on libqb version when
compile-time link is performed of the particular part":
X(a) .. ld.bfd < 2.29 OR [arbitrary ld.bfd AND libqb after this fix)
X(b) .. ld.bfd = 2.29 (and likely on) AND libqb up to, but excluding
this fix
* * *
Let's also state some imperfections and loops kept open:
Deficiencies:
* whenever anything is compiled against our install-time-modified
libqb.so so as to force the visibility of the discussed symbols
(or when compiling [with] libqb internally):
> /usr/bin/ld: warning: ../lib/qblog_script.ld contains output sections; did you forget -T?
- not solvable as long as we use the linker script, and there's
hardly any other way not requiring the libqb consumers to adapt
in any aspect
* as already mentioned, lacking compatibility with ld.gold linker and
won't foreseeably be (cf. https://bugzilla.redhat.com/1500898#c7)
- please stick with ld.bfd (i.e. default ld linker), which you
had to do in the past anyway (at least for compiling libqb
itself)
Open questions:
* should we enable attribute((__section__)) for powerpc and other minor
platforms if the feature is proved to be working there as well?
and if/when that's going to happen, we need to figure out the
transition plan to be spread throughout an extended period to keep
the transition smooth -- notably when now-with-callsite-section
clients will get run-time linked with callsite-section-not-a-default
libqb (say upon it's downgrade), and for that, the libqb's support
alone should be enabled year(s) ahead of the actual client space...
* * *
[*] basically GCC's section("SECNAME") __attribute__ annotation of the
global variables + linker described behaviour previously mistakenly
taken for granted
References:
[1] http://oss.clusterlabs.org/pipermail/developers/2017-July/000503.html
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1477354#c2 + comment 8
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1477354#c9
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
Now that the previous commit provides a foundation for what exactly can
go wrong with ld.bfd = 2.29+ linker, let's start reconciling that with
some reasonable assurance that logging is not silently severed, because
realizing the logs are missing is otherwise bound to happen when the
logs are suddently pretty crucial analytical resource :-)
We'll proceed in two steps as detailed.
* * *
As a first step, the table below concludes how the test matrix overview
introduced with a message for the preceding commit (also introducing
log_test_mock.sh runner which got reused here) looks as of this
refreshed sanity check, once QB_LOG_INIT_DATA macro at hand gets applied
(meaning "for non-libqb logging participants" so as not complicate the
matrix further). That macro is nothing triggered directly, it will just
plant a constructor-like function (to be invoked automatically early in
the execution) that will run through the checks (one original and couple
of new ones as of this changeset).
Note that for libqb users, this implies a new link dependency on libdl,
because they may opt-in for refreshed QB_LOG_INIT_DATA sanity check that
calls out to dlopen/dlsym/dladdr directly in case of "attribute section"
being available for the particular platform, and hence immediately needs
those symbols resolved in link time. Hence, add this conditional link
dependency to libqb.pc pkg-config file under Libs variable -- we
actually restore the occurrence of "-ldl" there as it used to be present
until commit 56754d0. While doing so, also move immediate link
dependencies of libqb (if any, currently not but that may be
a regression arising from the cleanup related to the mentioned commit)
represented with the LIBS autoconf variable under Libs.private variable
in libqb.pc, where it belongs per pkg-config documentation.
The promised table follows, but first as a recap, "X(Y)" denotes
"X linked with linker Y":
X(a) .. ld.bfd < 2.29
X(b) .. ld.bfd = 2.29 (and only 2.29)
goes like this (values in <angle brackets> denote non-trivial change
[not mere rewording] introduced as of this commit, in comparison to
the table stated in the preceding commit):
+=========+=========+=========+=========+=========+=========+=========+
#client(x)# libqb(a) usage # libqb(b) usage #
# vvv #---------+---------+---------+---------+---------+---------+
# V # direct | libX(a) : libX(b) # direct | libX(a) : libX(b) #
+=========+=========+=========+=========+=========+=========+=========+
# x = a # OK | OK : BAD[*2] #<BAD[*E]>|<BAD[*F]>:<BAD[*G]>#
# x = b # BAD[*A] | BAD[*B] : BAD[*C] #<BAD[*E]>|<BAD[*F]>:<BAD[*G]>#
+=========+=========+=========+=========+=========+=========+=========+
Woefully, nothing changes if we swap binutils 2.29 for 2.29.1, i.e.,
X(b) .. ld.bfd = 2.29.1, compared to previous state, i.e., the second
table from the previous commit is still applicable for that situation.
The added sanity checks are useful nonetheless, consider for example,
that attribute-section-less libqb is what gets run-time linked to an
attribute-section-full target. The most precise check we could use
-- a custom logger function applied in a self-test scheme -- is not
available at the point the macro-defined function gets invoked, simply
because qb_log_init hasn't been invoked by the time that constructor
gets triggered. However, what we can do is to add a non-trapping
libqb-residing reverse-testing of the client space that (and once it)
voluntarily initiates qb_log_init (delivering abruption all of a sudden
at some unanticipated, as opposed to a well-timed like with
constructors, execution point, seems pretty bad idea + libqb as
a library is a mere helper, not an undertaker :) -- this check then
only announces, via syslog (the only pre-enabled logging target),
the target's logging may be severed.
* * *
Hence, as a promised second step, after incorporating the syslog
change (and extending log_test_mock.sh so as to capture syslog
stream within the container), not much changes with the table above,
i.e., X(b) .. ld.bfd = 2.29:
[*A] in addition, unless QB_LOG_INIT_DATA used on client side,
syslog carries this notice:
"(libqb) log module hasn't observed target chain supplied callsite
section, target's and/or libqb's build is at fault, preventing
reliable logging (unless qb_log_init invoked in no-custom-logging
context unexpectedly, or target chain built purposefully without
these sections)"
logged by libqb proper
[*C] in addition, unless QB_LOG_INIT_DATA used on interlib side,
syslog carries this notice:
"(libqb) log module hasn't observed target chain supplied callsite
section, target's and/or libqb's build is at fault, preventing
reliable logging (unless qb_log_init invoked in no-custom-logging
context unexpectedly, or target chain built purposefully without
these sections)"
logged by libqb proper
[*E] in addition, unless QB_LOG_INIT_DATA used on client side,
syslog carries this warning:
"(libqb) log module has observed target chain supplied section
unpopulated, target's and/or libqb's build is at fault, preventing
reliable logging (unless qb_log_init invoked in no-custom-logging
context unexpectedly)"
logged by libqb proper
[*F] in addition, unless QB_LOG_INIT_DATA used on interlib side,
syslog carries this warning:
"(libqb) log module has observed target chain supplied section
unpopulated, target's and/or libqb's build is at fault, preventing
reliable logging (unless qb_log_init invoked in no-custom-logging
context unexpectedly)"
logged by libqb proper
[*G] in addition, unless QB_LOG_INIT_DATA used on interlib side,
syslog carries this warning:
"(libqb) log module has observed target chain supplied section
unpopulated, target's and/or libqb's build is at fault, preventing
reliable logging (unless qb_log_init invoked in no-custom-logging
context unexpectedly)"
logged by libqb proper
but desirably changes with "X(b) .. ld.bfd = 2.29.1" one
(DEP ~ "depends"):
+=========+=========+=========+=========+=========+=========+=========+
#client(x)# libqb(a) usage # libqb(b) usage #
# vvv #---------+---------+---------+---------+---------+---------+
# V # direct | libX(a) : libX(b) # direct | libX(a) : libX(b) #
+=========+=========+=========+=========+=========+=========+=========+
# x = a # OK | OK : DEP[*J] #<BAD[*M]>|<BAD[*M]>:<BAD[*L]>#
# x = b #<DEP[*N]>| DEP[*I] :<DEP[*O]>#<BAD[*M]>|<BAD[*M]>:<BAD[*L]>#
+=========+=========+=========+=========+=========+=========+=========+
* * *
[*1] client logging not working
[*2] interlib logging not working
[*3] both client and interlib logging not working
[*A] boils down to [*1], unless QB_LOG_INIT_DATA used on client side,
which then fails on
"implicit callsite section is populated, otherwise target's build
is at fault, preventing reliable logging"
assertion
[*B] boils down to [*1], unless QB_LOG_INIT_DATA used on interlib side,
which then fails on
"implicit callsite section is populated, otherwise target's build
is at fault, preventing reliable logging"
assertion
[*C] boils down to [*3], unless QB_LOG_INIT_DATA used on interlib side,
which then fails on
"implicit callsite section is populated, otherwise target's build
is at fault, preventing reliable logging"
assertion
[*E] boils down to [*1], unless QB_LOG_INIT_DATA used on client side,
which then fails on
"implicit callsite section is self-observable, otherwise target's
and/or libqb's build is at fault, preventing reliable logging"
assertion
[*F] boils down to [*3], unless QB_LOG_INIT_DATA used on interlib side,
which then fails on
"libqb's callsite section is populated, otherwise libqb's build is
at fault, preventing reliable logging"
assertion
[*G] boils down to [*3], unless QB_LOG_INIT_DATA used on interlib side,
which then fails on
"implicit callsite section is self-observable, otherwise target's
and/or libqb's build is at fault, preventing reliable logging"
assertion
[*I] boils down to [*1], unless QB_LOG_INIT_DATA used on client side,
which makes it, likely through self-reference keepalive (see
below), work OK
[*J] boils down to [*2], unless QB_LOG_INIT_DATA used on interlib side,
which makes it, likely through self-reference keepalive (see
below), work OK
[*K] boils down to [*3]
in addition, syslog carries this notice:
"(libqb) log module hasn't observed target chain supplied callsite
section, target's and/or libqb's build is at fault, preventing
reliable logging (unless qb_log_init invoked in no-custom-logging
context unexpectedly, or target chain built purposefully without
these sections)"
logged by libqb proper
[*L] boils down to [*3], unless QB_LOG_INIT_DATA used on interlib side
(sufficient?), which makes it, likely through self-reference
keepalive (see below), boil down just to [*1];
in addition, syslog carries this notice:
"(libqb) log module hasn't observed target chain supplied callsite
section, target's and/or libqb's build is at fault, preventing
reliable logging (unless qb_log_init invoked in no-custom-logging
context unexpectedly, or target chain built purposefully without
these sections)"
logged by libqb proper
[*M] boils down to [*1];
in addition, syslog carries this notice:
"(libqb) log module hasn't observed target chain supplied callsite
section, target's and/or libqb's build is at fault, preventing
reliable logging (unless qb_log_init invoked in no-custom-logging
context unexpectedly, or target chain built purposefully without
these sections)"
logged by libqb proper
[*N] boils down to [*M], unless QB_LOG_INIT_DATA used on client side,
which makes it, likely through self-reference keepalive (see
below), work OK
[*O] boils down to [*K], unless QB_LOG_INIT_DATA used on both client
and interlib side, which makes it, likely through self-reference
keepalive (see below), work OK (it's expected that this a mere
composite of situations [*I] and [*J] with consequences as stated)
* * *
Note: the only problematic (i.e. not captured automatically by the
QB_LOG_INIT_DATA macro presumably utilized at every non-libqb logging
system participant in the form of a discrete compilation unit)
combination with 2.29 is the one intersecting at "BAD[*2]" pertaining
"everything but interlib compiled with ld.bfd < 2.29". It would, of
course, be solvable as well, but presumably not in an easy way, and
that use case should not be as frequent.
Takeway: whenever your target (library or client program) actively
utilizes logging (meaning it emits at least a single log message,
otherwise there's an imminent danger of possibly even run-terminating
false positive in the self-check mechanism!),
YOU ARE strongly ENCOURAGED TO USE QB_LOG_INIT_DATA macro function
at (exactly) one of the source code files (presumably the main one)
per respective target's compilation unit.
It will alleviate the hassles possibly caused by downgrading libqb
to the linker-vs-libqb incompatibly compiled one or in similar
circumstances arising merely from the linker behaviour change,
which the current build system/code shake is all about.
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
...so as to evaluate use of resources. In particular, the intention
here is to uncover the observable differences between the same logging
code built with callsite section (default when available) and
purposefully (overriding that default by force) without it.
To allow for the latter being applied conveniently, new macro,
QB_KILL_ATTRIBUTE_SECTION, can be defined in the compile-time of the
client code that wishes to opt-out from the callsite section feature.
* * *
Following is a discussion on these differences, sticking with the
logging client code generated with the script defaults, i.e., as it
would be run with these switches:
--callsite-count=3640
--branching-factor=3
--callsites-per-fnc=10
--round-count=1000
and then built twice (as detailed in
tests/functional/log_external/Makefile.am):
* log_callsite_bench_sectionfull
- with callsite section
* log_callsite_bench_sectionless
- without callsite section, imposed with -DQB_KILL_ATTRIBUTE_SECTION
in CPPFLAGS
--> Static size of the executable:
$ size -B log_callsite_bench_section* | tr '\t' ' ' | tr -s ' ' \
| cut --complement -d' ' -f6 | column -t
> text data bss dec filename
> 82761 146180 4 228945 log_callsite_bench_sectionfull
> 190000 588 4 190592 log_callsite_bench_sectionless
We can see that sectionfull is few kB bigger in the object sections
of interest, though the text-data ratio changes considerably, with
code section being cut in half in comparison to sectionless, which
can actually help the code locality (and hence utilization
of CPU caches) in the former case.
--> Dynamic memory/heap operations:
$ valgrind --log-fd=1 ./log_callsite_bench_section{full,less} 2>/dev/null
total heap usage:
- log_callsite_bench_sectionfull:
> 88 allocs, 87 frees, 3,427 bytes allocated[*]
- log_callsite_bench_sectionless:
> 11,894 allocs, 11,893 frees, 486,035 bytes allocated[*]
[*] "32 bytes in 1 blocks still reachable" looks rather as a spurious
warning on the valgrind's side (matter of dynamic linking library)
Apparently, sectionless keeps stirring the heap constantly, with all
the possible downsides associated with that, like hitting the page
faults leading to less timely execution.
--> Run-time efficiency:
$ time ./log_callsite_bench_section{full,less} 2>/dev/null
mean attempts out of 3 consecutive runs:
- log_callsite_bench_sectionfull:
> real 0m1.298s
> user 0m0.965s
> sys 0m0.331s
- log_callsite_bench_sectionless:
> real 0m1.436s
> user 0m1.067s
> sys 0m0.365s
As expected, we can observe sectionfull is slightly faster/more
efficient.
* * *
Based on the above, we can conclude that leveraging the callsite
section for logging as facilitated by the toolchain intrinsics is
beneficial, especially for performance-critical applications (corosync
being the showcase here). Therefore it's desired to struggle for
retaining this nifty trick despite some troubles emerged with recent
binutils releases (starting with 2.29) and the changed behaviour we
relied on so far in respective ld.bfd linkers (as mentioned in
preceding commits). That motive is immediately followed -- well,
judging the impact fairly, actually outclassed -- with the intention
to preserve binary compatibility (incl. continuous library support for
callsite section offloading spread in the existing client space widely
for quite some years already) to the utmost extent possible.
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
These are for quick manual sanity checking, assuming the target audience
-- maintainers -- are clear on the context of use and the purpose
(perhaps with the help of static files for comparison and/or additional
checking harness, usually available through "make check", but not to be
confused with regular unit + broader tests). These test are meant to be
compiled on demand only, not during the standard building routine, for
which a trick leveraging GNUmakefile-Makefile precedence with GNU make
was devised (GNU make/gmake already required by configure script for
other reasons [some pattern-based matching not available with FreeBSD's
default "make", IIRC], so this introduces no new build dependency).
The respective new tests are meant to simulate logging variants in two
different library consumption models:
a. regular: linking against system-wide library
b. developmental: consuming library from a local sub-checkout tree,
using libtool conventions and hence attaching the
library through libqb.la intermediate library
descriptor of libtool
and between up to three possibly affected logging system participants
(discrete compilation units):
1. libqb itself will emit log messages in boundary conditions or
for tracing purposes
2. client program that consumes libqb's logging API directly
3. ditto, but the client program furthermore links with a library
(referred to as "interlib") that itself exercises the logging
API (it's also linked with libqb) -- through induction, this
should cover whole class of N interlib cases
Especially the latter perspective makes for a test matrix to possibly
(hopefully) demonstrate a fix allowing to cope with the changed
behaviour of ld from binutils 2.29+ wrt. boundaries denoting symbols for
a (custom) orphan section that are no longer externally visible. Such
commit is in the pipeline...
Developmental consumption model (a.) is now also tested automatically
in Travis CI runs and as a part of %check within upstream-suggested
libqb.spec for RPM packaging, whereas the regular one (b.) serves as
a building block for new log_test_mock.sh runner of said test matrix
-- it iterates through all the possible permutations of linker-imposed
implicit visibility of mentioned symbols between various affected
link participants all making use of logging (see 1. - 3. above) so as
to demonstrate A/ the impact of the problem (see table below), and
subsequently B/ that the fix is effective in all these situations
(updated table will be provided as well) once it lands. This script
also allows convoluting the test matrix further, notably with on-demand
defusing the self-checks based on QB_LOG_INIT_DATA macro, which is
of significance as demonstrated below (and will become even more
important with upcoming patches in this series).
* * *
Current state for such matrix, in which participants 1. - 3. map like:
1. ~ libqb(Y)
2. ~ "direct"
3. ~ libX(Y) [a.k.a. interlib]
and where "X(Y)" denotes "X linked with linker Y":
X(a) .. ld.bfd < 2.29
X(b) .. ld.bfd = 2.29 (and only 2.29),
goes like this:
+=========+=========+=========+=========+=========+=========+=========+
#client(x)# libqb(a) usage # libqb(b) usage #
# vvv #---------+---------+---------+---------+---------+---------+
# V # direct | libX(a) : libX(b) # direct | libX(a) : libX(b) #
+=========+=========+=========+=========+=========+=========+=========+
# x = a # OK | OK : BAD[*2] # BAD[*1] | BAD[*D] : BAD[*3] #
# x = b # BAD[*A] | BAD[*B] : BAD[*C] # BAD[*1] | BAD[*C] : BAD[*3] #
+=========+=========+=========+=========+=========+=========+=========+
whereas if we swap 2.29 for 2.29.1, i.e., X(b) .. ld.bfd = 2.29.1, we
can observe a somewhat simpler story (DEP ~ "depends"):
+=========+=========+=========+=========+=========+=========+=========+
#client(x)# libqb(a) usage # libqb(b) usage #
# vvv #---------+---------+---------+---------+---------+---------+
# V # direct | libX(a) : libX(b) # direct | libX(a) : libX(b) #
+=========+=========+=========+=========+=========+=========+=========+
# x = a # OK | OK : DEP[*J] # BAD[*1] | BAD[*1] : BAD[*L] #
# x = b # DEP[*I] | DEP[*I] : DEP[*K] # BAD[*1] | BAD[*1] : BAD[*L] #
+=========+=========+=========+=========+=========+=========+=========+
* * *
[*1] client logging not working
[*2] interlib logging not working
[*3] both client and interlib logging not working
[*A] boils down to [*1], unless QB_LOG_INIT_DATA used on client side,
which fails on 'implicit callsite section is populated' assertion
[*B] boils down to [*1], unless QB_LOG_INIT_DATA used on interlib side,
which fails on 'implicit callsite section is populated' assertion
[*C] boils down to [*3], unless QB_LOG_INIT_DATA used on interlib side,
which fails on 'implicit callsite section is populated' assertion
[*D] boils down to [*3], unless QB_LOG_INIT_DATA used on interlib side,
which makes it boil down just to [*1] (hypothesis: mere internal
self-reference to the section's boundary symbols makes them
overcome some kind of symbol garbage collection at the linkage
stage, so they are exposed even they wouldn't be otherwise as
demonstrated with the initial, plain case of [*3])
[*I] boils down to [*1], unless QB_LOG_INIT_DATA used on client side,
which makes it, likely through self-reference keepalive (see
below) work OK
[*J] boils down to [*2], unless QB_LOG_INIT_DATA used on interlib side,
which makes it, likely through self-reference keepalive (see
below) work OK
[*K] boils down to [*3], unless QB_LOG_INIT_DATA used on both client
and interlib side, which makes it, likely through self-reference
keepalive (see below) work OK (it's expected that this a mere
composite of situations [*I] and [*J] with consequences as stated)
[*L] boils down to [*3], unless QB_LOG_INIT_DATA used on interlib side
(sufficient?), which makes it, likely through self-reference
keepalive (see below) boil down just to [*1]
* * *
Note: as observed with [*D] case (libqb linked with ld.bfd < 2.29
whereas interlib and its client linked with ld.bfd = 2.29), the exact
availability of a working logging doesn't depend solely on the linkers
in question, but generally (further investigation out of scope) the
conclusion is that when 2.29 ld.bfd is involved somewhere in the chain
of logging-related discrete compilation units, also (self-)referencing
of the section's boundary denoting symbols is a deciding factor whether
particular logging source will be honored. This may be a result of
some internal linkage garbage collection mechanisms involved.
Anyway, it is supposed that the fix to broken-by-linkage logging can be
proclaimed complete once all combinations pass barring QB_LOG_INIT_DATA
usage (incurring the mentioned active referential use of the symbols),
along with a spin using it everywhere for good measure.
For another level of the analysis depth, one can further play with
combinations of -n{sc,cl,il} options (explained upon -h switch) to
log_test_mock.sh (taking an oracle, this is added mostly to justify
the upcoming self-check test change because linker-script-based
workaround for newer linkers will cause the section boundary symbols
to be present regardless if that section is utilized, leading to
a self-inflicted breakage due to these empty section symbols suddenly
winning in the symbol resolution mechanism).
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
There was an idea to make apparently run-time test set to that effect,
but it would make cross-building harder. So arrange the test as if it
would be meant for AC_TRY_RUN, but achieve the same as with the first
assertion by the means of inspecting the linked result with, possibly
target-specific, nm utility.
While arranging the test for AC_TRY_RUN, based on feedback by Ferenc
Wágner, unify and increase usability of the run-time error signalling
through assertions.
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
In cross-compilation, we can't run test programs, so configure just
bails out. Since there is no cache variable (e.g. ac_cv_blabla, we can't
even provide the correct result.
But in thise case, we don't really need to run to start with; we just
need to check if the toolchain headers know about CLOCK_MONOTONIC.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Using years-old benchmark attached to PostreSQL ML[1], I've observed
tiny bit more than double boost in speed when using fdatasync instead of
traditional fsync, on two Linux machines, each equipped with an SSD.
While the observation may be disputable (there are various
interpretations to what "synchronized I/O" actually means), by logical
extension of what the two are supposed to do, one can expect fdatasync
will perform no worse than fsync. Having the timestamps correct is
really not a priority, compared to timely processing of the message
stream. So let's use it whenever possible with QB_LOG_CONF_FILE_SYNC
requested.
As an aside, PostreSQL seems to be be using "fdatasync" method of
updating out to disk by default on Linux till today[2].
[1] https://www.postgresql.org/message-id/1095055866.414539fadb90d@webmail.rawbw.comhttps://www.postgresql.org/message-id/attachment/20659/syncbench.c
[2] https://www.postgresql.org/docs/current/static/runtime-config-wal.html#GUC-WAL-SYNC-METHOD
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
* IPC: Allow filesystem sockets to be chosen at run-time on Linux
Most of this patch came from Andrew Beekhof.
Keep a global variable that decides whether or not to use filesystem sockets
or abstract sockets for IPC connections. This variable is set by the presence of a file (default /etc/libqb/force-filesystem-sockets).
* tests: Fix test_ipcc_truncate_when_unlink_fails_shm test using FS sockets
When using filesystem sockets, the
test_ipcc_truncate_when_unlink_fails_shm test always fails, this was
because the unlink() call is wrapped to fail and so it never cleans up
the old version of the socket.
The fix is to preemptively remove the file before unlink gets wrapped.
* doc: Explain the force-filesystem-sockets option
Signed-off-by: Christine Caulfield <ccaulfie@redhat.com>
The intention behind enabling the syslog tests on demand (642f74d) was
to tread cautiously as libtool used to alert that something controversial
is going on:
$ ./autogen.sh && ./configure --enable-syslog-tests && make check
> ...
> *** Warning: Linking the executable log.test against the loadable module
> *** _syslog_override.so is not portable!
> ...
owing to the fact that _syslog_override.so is being linked (through
libtool) with LDFLAGS=-module.
In fact, the issue did go away with a fix to the "make install" process
(ebcff54) for which _syslog_override.so* files were being picked
undesirably prior to the fix, which manifested itself also on
"make rpm" (hence the respective commit message):
> RPM build errors:
> Installed (but unpackaged) file(s) found:
> /usr/lib64/_syslog_override.so
> /usr/lib64/_syslog_override.so.0
> /usr/lib64/_syslog_override.so.0.0.0
The explanation for the secondary positive effect on the mentioned
libtool's warning going away is rather simple:
"lib" prefix switches the expectations about the result as a library
(also due to "libdir" destination) rather than as an executable object.
Change like that ensures the result is what libtool's manual calls
"dynamic module that can be opened by lt_dlopen" rather than "runtime
library that cannot" in the context of Mac OS X, which was likely the
cause of the portability warning.
(https://www.gnu.org/software/libtool/manual/libtool.html#FOOT10)
And because the tests themselves are run through the libtool wrapper
scripts capable of relinking and other magic
What libtool manual has to say on this topic:
> Note that libtool modules don’t need to have a "lib" prefix. However,
> Automake 1.4 or higher is required to build such modules.
referring to version released 15+ years back, which is assumed anyway.
(https://www.gnu.org/software/libtool/manual/libtool.html#Modules-for-libltdl)
* * *
That being said, there's now no longer a reason to have "syslog-tests"
enablement conditionalized (no longer supposed intrusive), hence enable
them unconditionally along the other libcheck-based tests
-- whenever libcheck is detected.
This changeset also drops any trace of --enable-syslog-tests switch
in various places using that (spec file, Travis CI configuration)
making for a complete vaporizatio of this choice.
This reduces overlinking of qb-blackbox. Being a seldom used executable,
the gains are mostly theoretical, but at least this silences warnings
from some QA tools.
There are not many ways to test alternate code paths having failure of
some function from standard library as a precondition.
For a starter, we need to test failing unlink{,at} functions in a
controlled manner to mimic client and server path of the IPC connection
having different privileges to validate the previous commit. But the
test suite cannot assume it has root privileges (so as to add artificial
user system-wide, which is a pretty stupid idea on its own), cannot
generally use stuff like chroot/namespacing (not to speak about
synergies of the former like docker). So what's left is to make our
own playground, or better yet, use existing playground but just to
modify the rules of the game a bit when it's desired -- a variation
of old good LD_PRELOAD trick.
Note that this concept was already used in syslog tests (see commit
642f74d) and is now further extended using dlsym(RTLD_NEXT, "symbol")
to resolve the standard library symbol being shadowed by our little
"module". This hence yields a customized wrapping we use to either
inject a call failure or to increase an invocation counter so as to
assure something has indeed been called. As the mechanisms used are
not supposed to be available everywhere, the build system is
conditionalized respectively.
Back to our test when unlink{,at} fails, with the help of the described
mechanism, it was actually easy to massage test_ipc_server_fail_shm
into test_ipcc_truncate_when_unlink_fails_shm desired addition, which
is also featured in this commit, together with a modification to
resources.test script so that it expects particular number of empty
file leftovers (see previous commit).
It's expected that the module for failure injections will keep growing
so as to enable better overall coverage of the code (on the platforms
where this provision is available).
These are intended for subsequent qb_rb_{force_,}close refactorization
and utilization of this new truncate as a fallback after unlink failure
as detailed in the commit to follow.
For newer POSIX revision compliant systems, there's "at" variant using
openat/unlinkat functions so that paths do not have to be traversed
in full anew when not needed (as both unlink and truncate operate on
the same path).
Under the influence of libxml2 and considering that actual "patch"
information in fact, if significant, ends up encoded in QB_VER_REST,
shift away from convention codified, e.g., by semver.org (not adored
by libqb, anyway) and rename designated PATCH component of the version
to MICRO accordingly.
Note that at this point, after a release without any header-based
versioning present and just a few commits after it was tentatively
introduced, it's a painless change. Once this PATCH nomenclature
is leaked into a full release, there's no way to get rid of it
reasonably...
