Previously, when a valgrind memleak was discovered, would cause a
catastrophic pytest failure. Now correctly fails the current pytest as
intended.
As a result of this fix --valgrind-memleaks now works in distributed
pytest mode as well.
Signed-off-by: Christian Hopps <chopps@labn.net>
- Fix xterm support to work, previously it mostly didn't, not it should
in all cases (i.e., single or dist mode).
- Catch when the user tries to use various window requiring topotests
features (e.g., --cli-on-error) but isn't running under supported
system (e.g., byobu/tmux/xterm), and fail the run with an explanation.
Signed-off-by: Christian Hopps <chopps@labn.net>
Refactor the bgp_auth test to create common_config code to allow
non-json based tests to reset routers and load configs in parallel.
Signed-off-by: Christian Hopps <chopps@labn.net>
TMUX and Screen support when running topotests inside docker. This
allows the gdb, shell and vtysh features to correctly work even when
running the tests inside docker.
Add options:
--asan-abort :: aborts the process on ASAN errors
--strace-daemons :: strace some or all daemons
Signed-off-by: Christian Hopps <chopps@labn.net>
Fixes:
/usr/lib/python3.9/site-packages/_pytest/config/__init__.py:1463: in getoption
val = getattr(self.option, name)
E AttributeError: 'Namespace' object has no attribute 'topology_only'
The above exception was the direct cause of the following exception:
/usr/lib/python3.9/site-packages/pluggy/manager.py:127: in register
hook._maybe_apply_history(hookimpl)
/usr/lib/python3.9/site-packages/pluggy/hooks.py:333: in _maybe_apply_history
res = self._hookexec(self, [method], kwargs)
/usr/lib/python3.9/site-packages/pluggy/manager.py:93: in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
/usr/lib/python3.9/site-packages/pluggy/manager.py:84: in <lambda>
self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
tests/topotests/conftest.py:62: in pytest_configure
if config.getoption("--topology-only"):
/usr/lib/python3.9/site-packages/_pytest/config/__init__.py:1474: in getoption
raise ValueError(f"no option named {name!r}") from e
E ValueError: no option named 'topology_only'
Signed-off-by: Quentin Young <qlyoung@nvidia.com>
Diff'ing JSON objects is a crucial operation in the topotests for
comparing e.g. vtysh output (formatted as JSON) with a file which
covers the expectation of the tests. The current diff functionality
is 'self-written' and intended to test a JSON object d2 on being a
subset of another JSON object d1. For mismatches a diff is generated
based on a normalized textual representation of the JSON objects.
This approach has several disadvantages:
* the human provided JSON text might not be normalized, hence
a diff with line numbers might be worthless since it provides
close to zero orientation what the problem is
* the diff contains changes like commatas which are meaningless
* the diff might contain a lot of changes about meaningless
content which is present in d1 but not in d2
* there is no proper functionality to test for 'equality' of
d1 and d2
* it is not possible to test for order, e.g. JSON arrays are
just tested with respect to being a subset of another array
* it is not possible to check if a key exists without also
checking the value of that particular key
This commit attempts to solve these issues. An error report is
generated which includes the "JSON Path" to the problematic JSON
elements and also hints on what the actual problem is (e.g. missing
key, mismatch in dict values etc.).
A special parameter 'exact' was introduced such that equality can be
tested. Also there was a convention that absence of keys can be
tested using the key in question with value 'None'. This convention
is still honored such that full backwards compatiiblity is in
place.
Further order can be tested using the new tag '__ordered__' in
lists (as first element). Example:
d1 = [1, 2, 3]
d2 = ['__ordered__', 1, 3, 2]
Tesing d1 and d2 this way will now result in an error.
Key existence can now be tested using an asterisk '*'. Example:
d1 = [1, 2, 3]
d2 = [1, '*', 3]
d1 = {'a': 1, 'b': 2}
d2 = {'a': '*'}
Both cases will result now in a clean diff for d1 and d2.
Signed-off-by: GalaxyGorilla <sascha@netdef.org>
When an `assert` fails we should skip all other tests on the file. Once
a failure is detected we can't rely on the setup anymore, since most of
the tests assume the previous worked.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
As of pytest 4.something, a pattern we were using in conftest.py
was deprecated. Also make a new-ish test script executable (all
the rest appear to be?)
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Create a specialized assert and json_cmp() result to improve the
comparison output. With this we also got a way to display all comparison
failures instead of just the first one.
This allows old tests to be run with '--topology-only' without
generating tons of error messages, instead it will just stop the test
without trying anything else.
Topogen (Topology Generator) is a helper that wraps around Topotest to
simplify some of the boilerplate code. This abstraction will help the
development of new tests and new APIs without breaking the existing
ones. It also makes the relation of objects clearer, since we no longer
touch the Mininet API directly, which in turn also makes us less
vulnerable to external API changes.
