Instead of waiting for daemons start with `sleep`, start them with the
`-d` parameter so they can release the terminal themselves when ready.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Start logging early everything (including debug) to
`/tmp/topotest/<test>/<node>/<daemon>.{out,err}`.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Handle the duplicated code with a simple conditional: if called from
specialized API use provided daemons configuration, otherwise fallback
to old `Router` own daemon settings.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Include vrf name with interface name when topojson framework
generates interface configuration. This matches the output of
'show runn', and makes config reset less disruptive. Also
stop removing configured debugs and log output when re-generating
config.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Error Message seen:
2020-06-11 14:00:35,288 ERROR: assert failed at "test_ebgp_ecmp_topo2/test_ecmp_after_clear_bgp[redist_static]": Testcase test_ecmp_after_clear_bgp[redist_static] : Failed
Error: TIMEOUT!! BGP is not converged in 30 seconds for router r3
assert 'TIMEOUT!! BGP is not converged in 30 seconds for router r3' is True
if a retry for a failed connection is 120 seconds we should wait slightly
longer than a retry session, which this clear test was not doing.
Especially since we know our topotests are lossy on data under load.
Apparently I changed this earlier to 90 seconds, but a retry window
is 120. Not sure wtf I was thinking
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
The convergence timeout in lib/bgp.py was something like 40 secs;
that's not really long enough for the CI. Raise the timeout to
40 x 3 secs.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Just a simple setup for pbr to prove it starts. Once the json
code for pbr gets in we can add more.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
The involved piece of code is supposed to find a 'closest' match for two
JSON structures using another JSON diff. However, it can happen that
during that new diff the JSON structures are altered (elements from a
list are deleted when 'found'). This is in general ok when the deleted
element is part of the JSON structure which 'matches', but when it later
turns out that some other element of the structure doesn't fit, then the
whole structure should be recovered. This is now realized by using a
deepcopy for the besaid new JSON diff such that the original is only
altered (e.g. deleted) when the diff is clean.
Signed-off-by: GalaxyGorilla <sascha@netdef.org>
1. Test suite has 25 test cases to verify BGP-graceful-restart functionality
2. Execution time is ~21 Mins
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Adding APIs to common_config.py to support BGP-Graceful-Restart automation
2. Adding APIs to create BGP-GR config to bgp.py
3. Adding verification API for BGP-GR functionality
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
Diff'ing JSON objects is a crucial operation in the topotests for
comparing e.g. vtysh output (formatted as JSON) with a file which
covers the expectation of the tests. The current diff functionality
is 'self-written' and intended to test a JSON object d2 on being a
subset of another JSON object d1. For mismatches a diff is generated
based on a normalized textual representation of the JSON objects.
This approach has several disadvantages:
* the human provided JSON text might not be normalized, hence
a diff with line numbers might be worthless since it provides
close to zero orientation what the problem is
* the diff contains changes like commatas which are meaningless
* the diff might contain a lot of changes about meaningless
content which is present in d1 but not in d2
* there is no proper functionality to test for 'equality' of
d1 and d2
* it is not possible to test for order, e.g. JSON arrays are
just tested with respect to being a subset of another array
* it is not possible to check if a key exists without also
checking the value of that particular key
This commit attempts to solve these issues. An error report is
generated which includes the "JSON Path" to the problematic JSON
elements and also hints on what the actual problem is (e.g. missing
key, mismatch in dict values etc.).
A special parameter 'exact' was introduced such that equality can be
tested. Also there was a convention that absence of keys can be
tested using the key in question with value 'None'. This convention
is still honored such that full backwards compatiiblity is in
place.
Further order can be tested using the new tag '__ordered__' in
lists (as first element). Example:
d1 = [1, 2, 3]
d2 = ['__ordered__', 1, 3, 2]
Tesing d1 and d2 this way will now result in an error.
Key existence can now be tested using an asterisk '*'. Example:
d1 = [1, 2, 3]
d2 = [1, '*', 3]
d1 = {'a': 1, 'b': 2}
d2 = {'a': '*'}
Both cases will result now in a clean diff for d1 and d2.
Signed-off-by: GalaxyGorilla <sascha@netdef.org>
Modify the import-check command to require the underlying prefix
to exist in the rib. General consensus is that this is the correct
behavior.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
1. Added 2 new test cases to bgp-basic-functionality-topo1
2. Enhanced 2 tests to run for both static routes and network advvertise command
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Added 5 test cases to verify BGP AS-allow-in behavior in FRR
2. Enhanced framework to support BGP AS-allow-in config(lib/bgp.py)
3. Added API in bgp.py to verify BGP RIB table(lib/bgp.py)
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Added 1 test case to verify NO-ADVERTISE Community functionality
2. Enhanced bgp.py to exclude routers from verification, if doesn't have bgp config
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Added 5 test cases to verify BGP AS-allow-in behavior in FRR
2. Enhanced framework to support BGP AS-allow-in config(lib/bgp.py)
3. Added API in bgp.py to verify BGP RIB table(lib/bgp.py)
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Used aggresive values to verify keepalive and holddown timers functionality
2. Modified variable name in lib/bgp.py
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Modified scripts to optimize execution time for route-map test suite
2. Removed unnecessary print statements from /lib/bgp.py
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
This is necessary to make some tests work correctly. Only topologies
created using the topogen framework benefit from this change.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
Add cli error returned for invalid bgp-community-list command
Fix retry function to avoid retries when expected is False
The lib/bgp.py test code is bringing up neighbors and clearing them
to test that things are working appropriately. The problem we have
is that we are only waiting 30 seconds for declaration of failure.
In a high load system packets can be lost and as such the initial
convergence may not happen. Modify the test to wait for 1 retry
window test period before declaring failure.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
1. Add check if show running output is corrupted as frr-reload does
not return cause of failure, just the error codes.
2. Change logger level to debug for some extra information.
3. Modify logger messages for more clear information.
4. Print configuration commands to console instead of show running
5. Print show command output to console.
6. Move show running output within flag show_router_config.
7. Add retry decorator for retyring show commands.
The peerUptime data received from a `show bgp ipv4 uni summ json`
gives you the time in seconds since the peer has come up.
The clear_bgp_and_verify function is checking the peerUptime
for before and after and if they are the same then the test
was declaring a clear failure.
The problem with this is of course that the tests can run fast
enough that the peerUptime is the same for before and after the clear.
Modify the test case to use peerUptimeEstablishedEpoch.
This value is the seconds since the epoch since the establishment
of the peer. This will allow us to know that a clear happened.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
The new `run_and_expect` variant - called `run_and_expect_type` - tests
the return value type of the test function and optionally the return
value.
Now we can implement tests from test functions that return different
return types.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Attempt to run the function multiple times to make sure we got the
result we expected. This is a middle ground between big sleeps and short
test intervals.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Allow users of `luCommand` to specify wait time between commands
execution and simplify the `wait` method implementation.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Fix two main issues:
* Don't use float to figure out if we spent the time user asked;
* Don't depend on system clock to find we reached the end of time;
The fix is basically pre caculating the amount of wait cycles we are
going to peform and use a counter.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
All debug logging has been thrown to the floor in this loop.
Add it back in and additionally add a bit more debugging to
know what is going on.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
A bunch of our current tests setup data and redistribute
it across some bgp connections and then test for it
being there. A delay of 2 seconds that was initially
there to ensure that the data has propagated does not
actually work all the time when you have a loaded
virtualized environment.
Make the sleep 10 seconds. I agree this is not the ideal
solution but I would rather get the damn tests up and running
again.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
frr-reolad.py invocation sends info message to stderror channel
Redirected stderror from frr-reload.py to stdout
Fixed verify_convergence when more than one link is present
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
Creates logdir everytime in start_topology, removed stop_topoloy.
Using tgen.stop_topology in test case now.
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
Adds mulitple test scenarios for prefix-lists in bgp. Verfication
API for prefix-lits
Also adds description for bgp-basic test suite
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
Adding verify and config apis for bgp timer testcase
Adding verify and config apis for changing ASN to 4 bytes
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
Adding mulitple methods to form BGP configuration and other
helper methods.
If "bgp" is given in JSON the configuration will be created
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
Adds method to load configuration on the router. Save configuration
for resetting.
Adds method for configuring interfaces ip address on router.
Adds logdir and other pytest.ini option
Adding in the command `show log-filter` made `show log`
ambiguous. Change the checkRouterRunning() test to do
full `show logging` so it works again.
Signed-off-by: Stephen Worley <sworley@cumulusnetworks.com>
Add a common function to retrieve and canonicalize
'show ipv6 route' output for use in topotests. Use that in
the test_ospf6_topo1 test; update the corresponding 'expected'
results files to match the lib function.
Replace some 'print' with 'logger' statements in that test also.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Change the router log output to the previous folder so it doesn't get
erased when starting the old API (unbreaks command/output logging on
Topogen).
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
`param.get` always evaluates the second argument and it was causing
two log directories being created for topologies using Topogen.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Start the addition of FreeBSDRouter class that inherits from
class Router. At this point we do not do a whole lot. We
just come up on FreeBSD, lots of stuff that still needs to
be fixed.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
As part of the class rework, move the linux specific config
into the LinuxRouter.config section instead of being in the
Router class.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Modify the LinuxRouter code such that it inherits from the
Router class. This is setup work for (a) pulling out linux
specific config from class Router and (b) creating a FreebsdRouter
that inherits from class 'Router'.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
The sharp daemon did not have any ability to be used in
topotests. Add some code to allow this.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Start abstraction of the module_present into module_present_linux and freebsd
To allow for multiple platforms that the topotests can be run on.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Start the cleanup of diagnose_env to allow the running of topotests
on platforms besides linux.
So we split up diagnose_env into linux and freebsd variants.
At this point in time freebsd doesn't have any special code.
To be determined in the future.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Before this update, json_cmp_results which were formatted as strings
would not show the error mesage but just an object reference.
Signed-off-by: Christian Franke <chris@opensourcerouting.org>
Add a breadcrumb for people testing to hopefully allow
them to figure out what is going wrong when they
are testing different versions of FRR using
topotests and staticd is not running because
this version of staticd needs to be cleaned up
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Label proxy manager test covering both LDP and BGP label requests.
lib/topotest.py:restartRouter() function has been fixed in order to
allow propagating options to deamons other than zebra.
Signed-off-by: F. Aragon <paco@voltanet.io>
The code was not attempting to set the max platform_labels
on a per router basis, leaving us w/ situations where
mpls install would fail because platform_labels was 0.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
In the case where vrf backend is based on NETNS, the link enable command
switches from context so as to perform the command.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
In a effort to migrate more tests to use `run_and_expect` instead of
`sleep`s, this commit imports some common functions used to compare
router output. Retrying output comparison for N times, instead of
relying on arbitrary code `sleep`s, should help CI system tests fail
less and possibly run for less time.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
When normalizing a text also remove trailing whitespace since external
tools might add them. This commit fixes a test failure in ospf_topo1 on
Ubuntu 18.04.1.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Standardize run_and_expect to show start and end time along with maximum
estimated wait time.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Don't quit on the first match. While here add some more `json_cmp` tests
to make sure this doesn't happen again.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Allow json_cmp to compare JSON values which begin with a list instead of
a dictionary.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Prior behavior of luCommand was to convert newlines to spaces in DUT
output before pattern matching. New method operating in parallel uses
re.DOTALL to mimic same behavior and allow preserving original DUT output.
The original output is needed for some scripts that parse line-by-line.
There is also some test code to compare match results using the
old way and new way and log a message if they are different. After
some short time we can develop confidence that using this new method
will not break any existing tests.
Signed-off-by: G. Paul Ziemba <paulz@labn.net>
topotest library is being added a new parameter when preparing the
configurationof each daemon. This parameter will be used to call the
daemon with some extra parameters. The -n parameter for zebra is taken
into account.
Also, when a extra parameter is given for calling zebra, a check is done to
see if it is possible to run zebra daemon with that option. This is the
case for vrfwnetns option. If not available, an error message is sent
back.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
Use the pytest assert function to be able to report to pytest failures
that happened inside the API.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
