Create a pid file for the router created by topotest.
By executing nsenter directly against this pid, developers
can execute commands directly from outside the unet shell.
This allows the developer to use script, tab completion, etc.,
and improves efficiency.
Signed-off-by: Hiroki Shirokura <slank.dev@gmail.com>
Refactor the bgp_auth test to create common_config code to allow
non-json based tests to reset routers and load configs in parallel.
Signed-off-by: Christian Hopps <chopps@labn.net>
- In order to run tests in parallel the netns-based vrfs need to
have unique names primarily bc they are all tracked/looked-up in
`/run/netns` which is not network namespace nesting friendly
- use ip(8) exclusively rather than a mix of `ip` and `ifconfig`
and `vconfig`, reducing required pkg count by a couple.
Signed-off-by: Christian Hopps <chopps@labn.net>
- bugs in the support library function `verify_gr_address_family`
allowed this test to pass depending on ordering of python dictinoary
keys. Fix the bugs, fix the test.
Signed-off-by: Christian Hopps <chopps@labn.net>
- A more general fix for the bgp listener test which requires interfaces be
configured in the kernel when the bgpd daemons are launched.
Signed-off-by: Christian Hopps <chopps@labn.net>
- Remove incorrect requirement for `service integrated-vtysh-config`
when producing a delta.
- Add `--test-reset` option which suppresses non-parseable lines from the
produced delta
- Use new features in common_config.py
Signed-off-by: Christian Hopps <chopps@labn.net>
TMUX and Screen support when running topotests inside docker. This
allows the gdb, shell and vtysh features to correctly work even when
running the tests inside docker.
Add options:
--asan-abort :: aborts the process on ASAN errors
--strace-daemons :: strace some or all daemons
Signed-off-by: Christian Hopps <chopps@labn.net>
Using "write memory" to save the daemons' configurations before
restarting them can cause log files to stop working correctly. Add
a new "save_config" to the kill_router_daemons() function to prevent
that from happening when saving the configurations isn't necessary.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Speedup (large topo): OLD: ~6 minutes NEW: ~1 second
(when paired with generate_support_bundle.py changes)
- Collect from each node in parallel
Bug fixes:
- sub-directory test name was the same internal pytest function name
for any test, and not the actual test name.
Signed-off-by: Christian Hopps <chopps@labn.net>
This python fixture was way too complex for what is needed.
Eliminate gratuitous options/over-engineering:
- Change from non-deterministic `wait` and `attempts` to a single
`retry_timeout` value. This is both more deterministic, as well as
what the user should actually be thinking about.
- Use a fixed 2 second pause between executing the wrapped function
rather than a bunch of arbitrary choices of 2, 3 and 4 seconds
spread all over the test code.
- Get rid of the multiple variables for determining what "Positive" and
"Negative" results are. Instead just implement what all the user code
already wants, i.e., boolean False or a str (errormsg) means
"Negative" result otherwise it's a "Positive" result.
- As part of the above the inversion logic is much more comprehensible
in the fixture code (and more correct to boot).
Signed-off-by: Christian Hopps <chopps@labn.net>
Issue: There was an error reported by Pylint regarding "expected" keyword:
Unexpected keyword argument 'expected' in function call (unexpected-keyword-arg)
Fix: We have defined expected keyword in all topojson APIs.
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
The current log prints maximum wait time which is not actually correct,
because it doesn't include the command execution time. We usually have
"failed after X seconds" log with X being far longer than this maximum.
Let's print the maximum number of tries instead.
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
This function kills all processes that happen to have the same
name to frr processes and it was only ever used in the setup.
Setup should not be used to kill old runs. That should be a
separate process.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The following error is shown when running the OSPFv3 tests
2021-03-16 23:37:44,792 INFO: Function returned global name 'data_rid' is not defined
2021-03-16 23:37:44,792 INFO: Retry [#1] after sleeping for 2s
2021-03-16 23:37:46,794 INFO: Verifying OSPF6 neighborship on router r1:
2021-03-16 23:37:46,993 INFO: Output for command [ show ipv6 ospf6 neighbor ] on router r1:
Neighbor ID Pri DeadTime State/IfState Duration I/F[State]
2.2.2.2 1 00:00:03 Full/PointToPoint 00:00:01 r1-r2-eth0[PointToPoint]
Fix the "data_rid" warning by using the correct variable
Signed-off-by: ckishimo <carles.kishimoto@gmail.com>
Didn't test this but it's already randomly broken so cant be worse
Hopefully fixes:
raise InvalidCLIError("%s" % output)
InvalidCLIError: line 2: % Command incomplete[4]: bgp
large-community-list standard Test1 permit
Signed-off-by: Quentin Young <qlyoung@nvidia.com>
Pass the topogen 'tgen' object into the startRouterDaemons()
method. it can be used to start a debug cli immediately after
starting a daemon, and that can be handy.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
The changes add the "jsoncmp_pass" and the "jsoncmp_fail" commands to
compare VTY shell's JSON output to an expected JSON object during
topotests using the LabN testing framework. This helps to eliminate
false negative test results (i.e. due to routes beeing out of order
after convergence or cosmetic changes in VTY shell's text output).
Signed-off-by: David Schweizer <dschweizer@opensourcerouting.org>
When parsing the output of "ip -6 address", allow arbitrary base interface
names (the part after "@" in the interface name), not just "if0-9". Without
this, link-local addresses sometimes are attributed to the wrong interface
because we're not matching the interface name but still handle the
interface's addresses.
Signed-off-by: Martin Buck <mb-tmp-tvguho.pbz@gromit.dyndns.org>
Make the generate-support-bundle script and interactions more
python3-friendly, and use python3 explicitly.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Tests were timing out in our test system due to lost packets and
flakiness of the lower end systems. Just set the timers to 3/10
and give them plenty of time to converge.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
1. Adding api to verify ip nht command.
2. 5 cases of static routes with admin dist and tag
3. Run time = 89Secs
Signed-off-by: nguggarigoud <nguggarigoud@vmware.com>
Prepare the infrastructure to allow configuring and launching an SNMP
daemon as part of testing scenario.
Signed-off-by: Babis Chalios <babis@voltanet.io>
Signed-off-by: Pat Ruddy Chalios <pat@voltanet.io>
The bgp_gr_functionality_topo1 test was shutting down an
interface on r2 and then trying to bring it up on r1.
Hijinx ensued.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
1. Enhanced lib/topojson.py for creating topologies with switches and routers
2. Ran it through (black) for expected formatting
Signed-off-by: kuldeepkash <kashyapk@vmware.com>
1. Enhanced lib/common_config.py for common configuration/verification needed
for PIM automation
2. Ran it through (black) for expected formatting
Signed-off-by: kuldeepkash <kashyapk@vmware.com>
1. Added lib/pim.py for PIM configuration/verification
2. Ran it through (black) for expected formatting
Signed-off-by: kuldeepkash <kashyapk@vmware.com>
Timestamps in test logs are needed for correlation with messages in
routing protocol log files. Vox populi indicates preference for
timestamp at beginning of line.
OLD:
(#55) scripts/rip-show.py:61 COMMAND:r1:vtysh -c "show ip rip status": 00:0.* 00:0:wait:RIP Peers:
NEW:
Sat Dec 19 08:26:45 2020 (#55) scripts/rip-show.py:61 COMMAND:r1:vtysh -c "show ip rip status": 00:0.* 00:0:wait:RIP Peers:
Signed-off-by: G. Paul Ziemba <paulz@labn.net>
While accidently running the topotests with version 3
I keep getting:
TypeError: `dict_values` object does not support indexing..
version 2 of python dict.values() returns a list.
version 3 does not
Write some code to allow both to be handled.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
This new daemon manages Segment-Routing Traffic-Engineering
(SR-TE) Policies and installs them into zebra. It provides
the usual yang support and vtysh commands to define or change
SR-TE Policies.
In a nutshell SR-TE Policies provide the possibility to steer
traffic through a (possibly dynamic) list of Segment Routing
segments to the endpoint of the policy. This list of segments
is part of a Candidate Path which again belongs to the SR-TE
Policy. SR-TE Policies are uniquely identified by their color
and endpoint. The color can be used to e.g. match BGP
communities on incoming traffic.
There can be multiple Candidate Paths for a single
policy, the active Candidate Path is chosen according to
certain conditions of which the most important is its
preference. Candidate Paths can be explicit (fixed list of
segments) or dynamic (list of segment comes from e.g. PCEP, see
below).
Configuration example:
segment-routing
traffic-eng
segment-list SL
index 10 mpls label 1111
index 20 mpls label 2222
!
policy color 4 endpoint 10.10.10.4
name POL4
binding-sid 104
candidate-path preference 100 name exp explicit segment-list SL
candidate-path preference 200 name dyn dynamic
!
!
!
There is an important connection between dynamic Candidate
Paths and the overall topic of Path Computation. Later on for
pathd a dynamic module will be introduced that is capable
of communicating via the PCEP protocol with a PCE (Path
Computation Element) which again is capable of calculating
paths according to its local TED (Traffic Engineering Database).
This dynamic module will be able to inject the mentioned
dynamic Candidate Paths into pathd based on calculated paths
from a PCE.
https://tools.ietf.org/html/draft-ietf-spring-segment-routing-policy-06
Co-authored-by: Sebastien Merle <sebastien@netdef.org>
Co-authored-by: Renato Westphal <renato@opensourcerouting.org>
Co-authored-by: GalaxyGorilla <sascha@netdef.org>
Co-authored-by: Emanuele Di Pascale <emanuele@voltanet.io>
Signed-off-by: Sebastien Merle <sebastien@netdef.org>
1. Enhanced framework to
a. Verify fib active routes(lib/common_config.py).
b. Verify bgp multi path routes(lib/bgp.py).
c. Create mininet nodes with different names(lib/topojson.py).
4. 12 Test cases of static routing with ibgp.
Test suite execution time is ~30 minutes.
5. 12 Test cases of static routing with ebgp.
Test suite execution time is ~30 minutes.
Signed-off-by: naveen <nguggarigoud@vmware.com>
1. As per recent changes done in PR #7652, we have modified the auto-rd verification logic
2. Dev PR link: https://github.com/FRRouting/frr/pull/7652
Signed-off-by: kuldeepkash <kashyapk@vmware.com>
I accidently installed something that is telling me about
unlosed handles in the tests. Let's clean them up.
<and yes I have no idea wtf I did>
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
the topolog importation folder must be precised. otherwise following
error message appears:
root@dut-vm:~/topotests/bgp_flowspec# python3 test_bgp_flowspec_topo.py
Traceback (most recent call last):
File "test_bgp_flowspec_topo.py", line 96, in <module>
from lib.lutil import lUtil
File "/root/topotests/bgp_flowspec/../lib/lutil.py", line 25, in <module>
from topolog import logger
ImportError: No module named 'topolog'
root@dut-vm:~/topotests/bgp_flowspec#
The same error occurs with lutil and bgprib which are 2 libraries
located under lib/ folder. Some precisions are added too.
PR=71290
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
python3 does not support execfile implementation.
replace it with open and exec api that are available in both python 2
and 3 implementations.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
Fix lib to start loggin to correct daemon file on startup
Fix bgp-auth tests for the logging changes
Fixes Issue # 7545
Signed-off-by: Martin Winter <mwinter@opensourcerouting.org>
1) in generate_ips if start_ipaddr does not have a `/` in it
there exists a code path where both mask and step are
null values. Write a bit of code to ensure this pre-req
is found early and often
2) in verify_rib there exists a code path where static_route
is null when we get to the non static route section. Change
the code to operate on the advertise_network_dict that
we are iterating over.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Add support to compare the number of RIB nexthops, rather than the
specific nexthop addresses. Use this in the bgp_ecmp topotests that
test maximum-paths - testing the specific nexthops is wrong there,
it's not deterministic and we get spurious failures.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Ensure the list of daemons to start is either the one specified
by a caller or the default one from the router configuration.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
The linux kernel is getting RTM_F_TRAP and RTM_F_OFFLOAD for
kernel routes that have an underlying asic offload. Write the
code to receive these notifications from the linux kernel and
to store that data for display about the routes.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
1. Added 2 tests to verify bgp route aggregation using summary-only and
as-set commands
2. Execution time is ~90 sec
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Removed initial_wait in ospf library.
2. Removed one test case which was random
failure, will add back after traiging.
Signed-off-by: naveen <nguggarigoud@vmware.com>
1. Adding 18 ospf testcases to topojson.
2. Adding ospf.py library.
Test suite execution time is ~18 minutes.
Signed-off-by: naveen <nguggarigoud@vmware.com>
We have a bunch of tests that wait *then* check a command for success/failure.
Modify the tests to check *first* then to wait. This reduces test
run times on my system by ~1400 seconds for a full run.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
1. Added test to verify bgp vrf dynamic route leak functionality
2. Total execution time is ~8 mins
3. Added kernel version check, these script would be run for kernel version >= 4.19
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Topotest for isis-vrf is added for ipv4 and ipv6.
2. Test case for checking isis topology.
3. Test case for checking zebra isis routes.
4. Test case for checking linux vrf routes.
5. 2 new API's written in topotest/lib for checking vrf routes.
Co-authored-by: Kaushik <kaushik@niralnetworks.com>"
Signed-off-by: harios_niral <hari@niralnetworks.com>
Avoid unnecessary use of StringIO in one place, use version-
dependent method in another. Remove a couple of other py2->py3
problems.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Change the public router stop method to always do a two-phase
shutdown - once without waiting and a second time with a wait.
Ordinary callers need to use this approach when stopping routers.
Move the detailed internal details to a private method that tests
should not call directly.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
The vxlan `ip... ` command is failing because we are passing in
`no learning` and that is failing.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
When the topotest mem-leak reporting is enabled, use the same
two-step daemon stop procedure that's used in
the topogen.stop_topology path.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
add thread info, use "bt full" to get variables and add a bit of
disassembly for good measure.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
1. Increasing BGP convergence wait time to overcome Ubuntu 16.04 arm8 box, as
bgp neighorship is taking more time in this particular testbed.
2. Debugged bgp-ecmp-topo2 failures and here also it seems to be bgp convergence
issue, doing some enhancement in scripts to handle it
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
Stop printing hard-coded 30 seconds in a couple of places in
bgp.py in the topojson infra - print the actual time
spent waiting.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
1. It will generate support bundle/sump data on test failures
2. It used /usr/lib/frr/generate_support_bundle.py utility to dump the data
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
Another short timeout for bgp - make the
verify_bgp_convergence_from_running_config() api use the same
generous timeout as verify_bgp_convergence()
Signed-off-by: Mark Stapp <mjs@voltanet.io>
1. Added 7 test cases to verify bgp recursive nexthop and ebgp multi-hop functionality
2. Added framework support to automate these test cases
3. Total execution time is ~5 mins
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
Use the right list of daemons to avoid trying to start zebra twice.
Change a zebra log message to INFO level to avoid stderr check
failure.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Add a few retries during router shutdown before killing a daemon. Also
work harder to start only a single instance of daemons, esp. zebra.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Instead of waiting for daemons start with `sleep`, start them with the
`-d` parameter so they can release the terminal themselves when ready.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Start logging early everything (including debug) to
`/tmp/topotest/<test>/<node>/<daemon>.{out,err}`.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Handle the duplicated code with a simple conditional: if called from
specialized API use provided daemons configuration, otherwise fallback
to old `Router` own daemon settings.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Include vrf name with interface name when topojson framework
generates interface configuration. This matches the output of
'show runn', and makes config reset less disruptive. Also
stop removing configured debugs and log output when re-generating
config.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Error Message seen:
2020-06-11 14:00:35,288 ERROR: assert failed at "test_ebgp_ecmp_topo2/test_ecmp_after_clear_bgp[redist_static]": Testcase test_ecmp_after_clear_bgp[redist_static] : Failed
Error: TIMEOUT!! BGP is not converged in 30 seconds for router r3
assert 'TIMEOUT!! BGP is not converged in 30 seconds for router r3' is True
if a retry for a failed connection is 120 seconds we should wait slightly
longer than a retry session, which this clear test was not doing.
Especially since we know our topotests are lossy on data under load.
Apparently I changed this earlier to 90 seconds, but a retry window
is 120. Not sure wtf I was thinking
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
The convergence timeout in lib/bgp.py was something like 40 secs;
that's not really long enough for the CI. Raise the timeout to
40 x 3 secs.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Just a simple setup for pbr to prove it starts. Once the json
code for pbr gets in we can add more.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
The involved piece of code is supposed to find a 'closest' match for two
JSON structures using another JSON diff. However, it can happen that
during that new diff the JSON structures are altered (elements from a
list are deleted when 'found'). This is in general ok when the deleted
element is part of the JSON structure which 'matches', but when it later
turns out that some other element of the structure doesn't fit, then the
whole structure should be recovered. This is now realized by using a
deepcopy for the besaid new JSON diff such that the original is only
altered (e.g. deleted) when the diff is clean.
Signed-off-by: GalaxyGorilla <sascha@netdef.org>
1. Test suite has 25 test cases to verify BGP-graceful-restart functionality
2. Execution time is ~21 Mins
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Adding APIs to common_config.py to support BGP-Graceful-Restart automation
2. Adding APIs to create BGP-GR config to bgp.py
3. Adding verification API for BGP-GR functionality
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
Diff'ing JSON objects is a crucial operation in the topotests for
comparing e.g. vtysh output (formatted as JSON) with a file which
covers the expectation of the tests. The current diff functionality
is 'self-written' and intended to test a JSON object d2 on being a
subset of another JSON object d1. For mismatches a diff is generated
based on a normalized textual representation of the JSON objects.
This approach has several disadvantages:
* the human provided JSON text might not be normalized, hence
a diff with line numbers might be worthless since it provides
close to zero orientation what the problem is
* the diff contains changes like commatas which are meaningless
* the diff might contain a lot of changes about meaningless
content which is present in d1 but not in d2
* there is no proper functionality to test for 'equality' of
d1 and d2
* it is not possible to test for order, e.g. JSON arrays are
just tested with respect to being a subset of another array
* it is not possible to check if a key exists without also
checking the value of that particular key
This commit attempts to solve these issues. An error report is
generated which includes the "JSON Path" to the problematic JSON
elements and also hints on what the actual problem is (e.g. missing
key, mismatch in dict values etc.).
A special parameter 'exact' was introduced such that equality can be
tested. Also there was a convention that absence of keys can be
tested using the key in question with value 'None'. This convention
is still honored such that full backwards compatiiblity is in
place.
Further order can be tested using the new tag '__ordered__' in
lists (as first element). Example:
d1 = [1, 2, 3]
d2 = ['__ordered__', 1, 3, 2]
Tesing d1 and d2 this way will now result in an error.
Key existence can now be tested using an asterisk '*'. Example:
d1 = [1, 2, 3]
d2 = [1, '*', 3]
d1 = {'a': 1, 'b': 2}
d2 = {'a': '*'}
Both cases will result now in a clean diff for d1 and d2.
Signed-off-by: GalaxyGorilla <sascha@netdef.org>
Modify the import-check command to require the underlying prefix
to exist in the rib. General consensus is that this is the correct
behavior.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
1. Added 2 new test cases to bgp-basic-functionality-topo1
2. Enhanced 2 tests to run for both static routes and network advvertise command
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Added 5 test cases to verify BGP AS-allow-in behavior in FRR
2. Enhanced framework to support BGP AS-allow-in config(lib/bgp.py)
3. Added API in bgp.py to verify BGP RIB table(lib/bgp.py)
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Added 1 test case to verify NO-ADVERTISE Community functionality
2. Enhanced bgp.py to exclude routers from verification, if doesn't have bgp config
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Added 5 test cases to verify BGP AS-allow-in behavior in FRR
2. Enhanced framework to support BGP AS-allow-in config(lib/bgp.py)
3. Added API in bgp.py to verify BGP RIB table(lib/bgp.py)
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Used aggresive values to verify keepalive and holddown timers functionality
2. Modified variable name in lib/bgp.py
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
1. Modified scripts to optimize execution time for route-map test suite
2. Removed unnecessary print statements from /lib/bgp.py
Signed-off-by: Kuldeep Kashyap <kashyapk@vmware.com>
This is necessary to make some tests work correctly. Only topologies
created using the topogen framework benefit from this change.
Signed-off-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Signed-off-by: Ashish Pant <ashish12pant@gmail.com>
Add cli error returned for invalid bgp-community-list command
Fix retry function to avoid retries when expected is False
