There exist systems that do not explicity have a python soft-link
on their system. Let's explicity call out which python we want
to be using with exabgp.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The bgp gr topotests had run times that were greater than 10 minutes each.
Just brute force break up the tests to 4 different sub parts.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Description:
- Changing the expected output for selected route in the script.
- With our changes for VRF-Lite fix best path selection,
during best path selection, while comparing the paths for imported routes,
we should correctly refer to the original route i.e. the ultimate path.
In this case, when we have ibgp route and imported ibgp route
for the same prefix, we do compare IGP metric which is same for both,
So we proceed to comparing router-ids and selecting the best path.
- Before our changes, ibgp route was preferred because of IGP metric.
With our fix, expected output for selected route is changed to
imported ibgp route because of the lower router-id.
- Corresponding changes for expected advertised route and
the large community are made.
Co-authored-by: Kantesh Mundaragi <kmundaragi@vmware.com>
Signed-off-by: Iqra Siddiqui <imujeebsiddi@vmware.com>
Somewhere along the line core-files stopped being generated
with the running of the topotests. With this change we now
see this:
sharpd@eva /t/topotests> find . -name '*.dmp' -print
./ospfv3_basic_functionality.test_ospfv3_asbr_summary_topo1/r0/ospf6d_core-sig_6-pid_430478.dmp
sharpd@eva /t/topotests> sudo gdb /usr/lib/frr/ospf6d ./ospfv3_basic_functionality.test_ospfv3_asbr_summary_topo1/r0/ospf6d_core-sig_6-pid_430478.dmp
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/frr/ospf6d...
[New LWP 430478]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/ospf6d --log file:ospf6d.log --log-level debug -d'.
Program terminated with signal SIGABRT, Aborted.
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
(gdb)
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
This is implicitly checked by the "verify mroute" below, but it's much
more helpful to explicitly check in advance.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
Currently I get bgp_instance_del-test as well as bgp_l3vpn_to_bgp_vrf
failures every ~3-4 runs when under a 40 parallel run with micronet.
Examination of the failure and passing cases always leads to the
failures showing convergence of bgp bestpath immediately after
the show commands to ensure that the routes are there.
Modify the code to look for the fact that the vrf has
converged from routes being passed around across vrf's
and ensure that bestpath has run on them.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
When debugging issues for routes in multiple vrf's. It would
be extremely useful if the debug output had which vrf we
are acting on.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Because this test can be run in either netns vrf mode or vrflite
vrf mode, the default vrf name has different name. When netns mode
is chosen, vrf0 name is chosen as default name, while when vrflite
mode is chosen, default name is chosen. Remove the vrf keyword from
the expected dump.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
Issue #9983 explains what is wrong with the GR helper mode.
To unblock the CI that fails almost all the time on the ospf_gr_topo1
test, remove the commands and disable the test. Also add a reminder to
completely remove the helper mode if no one fixes the code in a month.
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
This can't really be run as part of CI, it's intended as a helper
instead, to use manually after poking around in the c-ares binding code.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
common_cli.c disables logging by default so stdio is usable as vty
without log messages getting strewn inbetween. This the right thing for
most tests, but not all; sometimes we do want log messages.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
949aaea5 removed debugs from all topotests, but this test relies on the
debug logs so it constantly fails now.
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
When our CI test system is under high load, expecting bfd to
converge in under 2 seconds is not going to happen. Modify the test
suites to just ensure that things reconvderge.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Debugs take up a significant amount of cpu time as well as
increased disk space for storage of results. Reduce test
over head by removing the debugs, Hopefully this helps
alleviate some of the overloading that we are seeing in
our CI systems.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The test system under load looks for upstream state only
1 time immediately after sending 2 streams of S,G data
flowing. Give the system some time to process this
and ensure that it actually shows up in a small
amount of time.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The test_ldp_pseudowires_after_link_down test
shuts a link down and was blindly waiting 5 seconds
before just assuming the test system was in a sane
state. Remove the sleep(5) and actually look for
the changed state for the route 2.2.2.2 that the
psueudowire actually depends on.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The test does this:
a) shut link down
b) test for ospf convergence
c) ensure the route is installed
When under a heavily loaded system c) is not guaranteed
to happen quickly. Give the system 10 extra seconds
to ensure it happens.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The route replace test was doing this seq of events:
a) Create nhg
b) Install route w/ sharpd
c) Ensure it worked
d) Modify nhg
d) Ensure the update group replace worked
The problem is that the sharp code is doing this:
/* Only send via ID if nhgroup has been successfully installed */
if (nhgid && sharp_nhgroup_id_is_installed(nhgid)) {
SET_FLAG(api.message, ZAPI_MESSAGE_NHG);
api.nhgid = nhgid;
} else {
for (ALL_NEXTHOPS_PTR(nhg, nh)) {
api_nh = &api.nexthops[i];
zapi_nexthop_from_nexthop(api_nh, nh);
i++;
}
api.nexthop_num = i;
}
The created nhg has not been successfully installed( or at least
sharpd has not read the results yet) when it gets the command
to install the routes. As such it passes down the individual
nexthops instead. The route replace is never going to work.
Modify the code to add a bit of sleep to allow sharpd to
get notified when the system is under load. At this point
there is no way to query sharpd for whether or not it
thinks it's nhg is installed properly or not. This
test is failing all over the place for a bunch of people
let's get this fixed so people can get running
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
the test_nexthop_groups function is failing occassionally
because the test executes 4 in succession sharp install
routes commands. When I dumped the rib on a failed test
run there were only 2 of the 4 routes in the rib and
the two that were in were the last 2 installed.
The sharp daemon setups a event process where it
installs routes `automatically`. If the previous
run is not finished entering a new command to install
the routes will mess up the last one from ever happening.
It is assumed that the user doesn't do stupid stuff here.
In this case I am just adding a small sleep between each
installation to just let the test proceed.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
the isis_topo1 test has two functions where immediately
after the test ensures that the routes are in isis
tests to see if they are in the rib. Under system
load I am seeing this test failing because the
routes are still queued. Modify the zebra check
for the isis routes to look for the proper results
for 10 seconds.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Currently, we have a lot of checks in CLI and NB layer to prevent
incompatible IS-types of circuits and areas. All these checks become
completely meaningless when the interface is moved between VRFs. If the
area IS-type is different in the new VRF, previously done checks mean
nothing and we still end up with incorrect circuit IS type. To actually
prevent incorrect IS type, all checks must be done in the processing
code.
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
This code has two issues:
a) The loop to test for successful installation re-installs
the route every time it loops. A system under load will
have issues ensuring the route is installed and repeated
attempts does not help
b) The nexthop group installation was always failing
but never noticed (because of the previous commit)
and the test was always passing, when it should
have never passed.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The test is checking installing of seg6 routes by this
loop:
for up to 5 times:
sharp install seg6 route
show ip route and is it installed
The problem is that if the system is under heavy
load the installation may not have happened yet
and by immediately reinstalling the same route
the same thing could happen again.
Modify the code to pull the route installation
outside of the loop and to increase to 10 attempts
in case there is very heavy system load.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The check_ping function `_check` function was asserting and being
passed to the topotests.run_and_expect() functionality causing
it to not run the full range of pings if one failed the test.
So effectively it was properly detecting pass / failure but
only allowing for 1 iteration if it was going to fail.
Modify the code to not assert and act like all the other
run_and_expect functionality.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
isis_tlvs.c would fail at multiple places if incorrect
TLVs were received in unpack_item_ext_subtlvs(),
causing stream assertion violations.
Signed-off-by: Juraj Vijtiuk <juraj.vijtiuk@sartura.hr>
The bgp_l3vpn_to_direct test is failing sometimes because
the 2.2.2.2 route is dissapearing. What is happening?
The log file for the failed test run shows us this:
build 15-Oct-2021 07:26:12 scripts/adjacencies.py:8 WAIT:r4:ping 2.2.2.2 -c 1: 0. packet loss:wait:PE->P2 (loopback) ping:60:0.5:
build 15-Oct-2021 07:26:12 Fri Oct 15 14:26:12 2021 (#9) scripts/adjacencies.py:8 COMMAND:r4:ping 2.2.2.2 -c 1: 0. packet loss:wait:PE->P2 (loopback) ping:
build 15-Oct-2021 07:26:12 COMMAND OUTPUT:PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data.
build 15-Oct-2021 07:26:12 64 bytes from 2.2.2.2: icmp_seq=1 ttl=64 time=0.143 ms
build 15-Oct-2021 07:26:12
build 15-Oct-2021 07:26:12 --- 2.2.2.2 ping statistics ---
build 15-Oct-2021 07:26:12 1 packets transmitted, 1 received, 0% packet loss, time 0ms
build 15-Oct-2021 07:26:12 rtt min/avg/max/mdev = 0.143/0.143/0.143/0.000 ms:
build 15-Oct-2021 07:26:12 Done after 1 loops, time=0.024507761001586914, Found= 0% packet loss
build 15-Oct-2021 07:26:12 Fri Oct 15 14:26:12 2021 (#9) scripts/adjacencies.py:9 COMMAND:r4:ping 2.2.2.2 -c 1: 0. packet loss:pass:PE->P2 (loopback) ping +0.02 secs:
build 15-Oct-2021 07:26:12 2021-10-15 14:26:12,446 WARNING: topolog.r4: LinuxNamespace(r4): proc failed: rc 2 pid 28826
build 15-Oct-2021 07:26:12 args: /usr/bin/nsenter -a -t 27444 -F --wd=/tmp/topotests/bgp_l3vpn_to_bgp_direct.test_bgp_l3vpn_to_bgp_direct/r4 /bin/bash -c ping 2.2.2.2 -c 1
build 15-Oct-2021 07:26:12 stdout: connect: Network is unreachable:
build 15-Oct-2021 07:26:17 COMMAND OUTPUT:connect: Network is unreachable:
build 15-Oct-2021 07:26:17 R:9 r4 PE->P2 (loopback) ping +0.02 secs 0 1
So the 2.2.2.2 route is coming/going and is failing on these test lines:
luCommand(
"r1", "ping 2.2.2.2 -c 1", " 0. packet loss", "wait", "PE->P2 (loopback) ping", 60
)
luCommand(
"r3", "ping 2.2.2.2 -c 1", " 0. packet loss", "wait", "PE->P2 (loopback) ping", 60
)
luCommand(
"r4", "ping 2.2.2.2 -c 1", " 0. packet loss", "wait", "PE->P2 (loopback) ping", 60
)
So the 2.2.2.2 routes on r1,3 and 4 are received via ospf, but are
modified by some other process to add labels ( probably ldp, since
it is running too ). The 2nd ping to 2.2.2.2 is failing because
the 2.2.2.2 route on r4 is being replaced. As an example here
is `ip monitor all` on r4 during boot up. Please note timestamps
are not necessarily representative of what we will see on the
loaded ci system.
[2021-10-15T15:46:52.261456] [NEXTHOP]id 27 via 10.0.2.2 dev r4-eth0 scope link proto zebra
[2021-10-15T15:46:52.261490] [ROUTE]2.2.2.2 nhid 27 via 10.0.2.2 dev r4-eth0 proto ospf metric 20
<snip>
[2021-10-15T15:46:53.556405] [NEXTHOP]Deleted id 27 via 10.0.2.2 dev r4-eth0 scope link proto zebra
<snip>
[2021-10-15T15:46:53.566575] [NEXTHOP]id 32 via 10.0.2.2 dev r4-eth0 scope link proto zebra
[2021-10-15T15:46:53.566585] [ROUTE]2.2.2.2 nhid 32 via 10.0.2.2 dev r4-eth0 proto ospf metric 20
For a small amount of time the route was *gone*. I believe the upstream
CI system hits that window sometimes, causing the test to fail.
This patch attempts to ensure that the 2.2.2.2 route should be learned
appropriately ( thus slowing it down ) before the test moves onto
the ping. I suspect the long term answer might be to add a test to
the scripts/adjancies.py script to ensure that the test does not
continue until the appropriate label is in place, but I want to
make the test run a bit more perscriptive in what it is looking
for here.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Recent commit 83f325901a had a accidental
turn of a 1 second wait into a 10 second wait
between retries. 10 seconds is too long.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Test doesn't wait long enough when it checks the routers after
restart. On slower systems, it frequently failed as it ran out
of time
Signed-off-by: Martin Winter <mwinter@opensourcerouting.org>
When our ci test system is under high load, expecting bfd to converge
in under 2 seconds is not going to happen. Modify the test suites
to just ensure that things converge. If we need actual functional
testing of bfd response times the topotests are not an appropriate place
to do this or we need to modify the test system to gather the data for
how long it takes after the tests are run.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
During a local CI run, bgp_ecmp_topo3 was failing
to properly notice the fast-convergence command
issued before the interface is shut down. As
such there exists a race condition where under
high load the zebra process can actually shut
an interface down before we have properly ensured
that fast convergence is on for ibgp.
Modify the test for in two ways:
1) Ensure that previous section makes sure
that we have properly converged for when we
bring back up the interfaces instead of
assuming that we have done so.
2) After issuing the fast-convergence command.
Ensure that bgp has fully processed it and is
ready to receive the interface down events
as triggers for shutting down the ibgp session.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
On a local CI run. The test_ldp_topo1.py showed fail to converge
on r3. r3 has 2 neighbors but only 1 was up when we got to
further steps in the test suites.
Modify the neighbor checking to `know` how many neighbors
should be operational and continue looking for them until
they are up and running.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Previously, when a valgrind memleak was discovered, would cause a
catastrophic pytest failure. Now correctly fails the current pytest as
intended.
As a result of this fix --valgrind-memleaks now works in distributed
pytest mode as well.
Signed-off-by: Christian Hopps <chopps@labn.net>
Revert the accidental enabling of the optional memleak tests that came
with the large micronet changeset.
Signed-off-by: Christian Hopps <chopps@labn.net>
The nexthop group code is installing routes and nexthop groups
and immediately expecting zebra to have processed the results
as a result there is a situation when the CI system is under
intense load that the nexthop group might not have been processed.
Add a bit of code to allow the test to give FRR some time
to finish work before declaring it not working.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
When the CI system is heavily loaded, we might see the following failures:
```
test failed at "test_config_timing/test_static_timing": assert 20.083204 <= 19.487716
```
Currently we allow each step to run 2 times slower than the initial
measurement. Let's allow them to run 3 times slower.
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
On the first step, the test creates 10000 static routes. It passes 10000
to `get_ip_networks` and it generates 10000 /22 routes.
On the fourth step, the test tries to remove 5000 previously created
routes. It passes 5000 to `get_ip_networks` and here starts the problem.
Instead of generating 5000 /22 routes, it generates 5000 /21 routes. And
the whole step is a no-op, we constantly see the following logs:
```
% Refusing to remove a non-existent route
```
To consistently generate same routes, `get_ip_networks` must always use
the same prefix length.
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
Our topotests send SIGBUS 2 seconds after a SIGTERM is
initiated. This is bad because under a heavily loaded
topotest system we may have a case where the system has
not had a chance to properly shut down the daemon.
Extend the time greatly before topotests send SIGBUS.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
This removes a giant `switch { }` block from lib/zclient.c and
harmonizes all zclient callback function types to be the same (some had
a subset of the args, some had a void return, now they all have
ZAPI_CALLBACK_ARGS and int return.)
Apart from getting rid of the giant switch, this is a minor security
benefit since the function pointers are now in a `const` array, so they
can't be overwritten by e.g. heap overflows for code execution anymore.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
*_anywhere(item) returns whether an item is on _any_ container. Only
available for unsorted containers for now.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
This provides a "is this item on this list" check, which may or may not
be faster than using *_find() for the same purpose. (If the container
has no faster way of doing it, it falls back to using *_find().)
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
Even if it doesn't matter for an unit test in general, it hides actual
leaks in the code being tested. Fix so any leaks will be actual bugs.
(Currently there aren't any, yay.)
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
This script is failing occassionally in our upstream topotests.
Where it was changing route-maps and attempting to see if
summarization was working correctly. The problem was that
the code appeared to be attempting to add route-maps to
redistribution in ospf then modifying the route-maps behavior
to affect summarization as well as the metric type of that
summarization.
The problem is of course that ospf does not appear to modify
the summary routes metric-type when the components
of that summary change it's metric-type. So the test
is testing nothing. In addition the test had messed
up the usage of the route-map generation code and all
the generated config was in different sequence numbers
but route-map processing would never get to those
new sequence numbers because of how route-maps are processed.
Let's just remove this part of the test instead of trying
to unwind it into anything meaningfull
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Several tests used the route_map_create functionality
with `metric-type` but never bothered to add the
backend code to ensure it works correctly.
Add it in so it can be used.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
We have this pattern in this test:
# Let's kill the interface on rt2 and see what happens with the RIB and BFD on rt1
tgen.gears["rt2"].link_enable("eth-rt1", enabled=False)
# By default BFD provides a recovery time of 900ms plus jitter, so let's wait
# initial 2 seconds to let the CI not suffer.
topotest.sleep(2, 'Wait for BFD down notification')
router_compare_json_output(
"rt1", "show ip route ospf json", "step3/show_ip_route_rt2_down.ref", 1, 0
)
Under a heavy CI load, interface down events and then reacting to them may not actually
happen within 2 seconds. Allow some more grace time in the test to ensure that we
react to it in an appropriate manner.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
OSPF when it is deciding on whom it should elect for DR and backup
has a process that prioritizes network stabilty over the exact
same results of who is the DR / Backups.
Essentially if we have r1 ----- r2
Let's say r1 has a higher priority, but r2 comes up first, starts
sending hello packets and then decides that it is the DR. At some
point in time in the future, r1 comes up and then connects to r2
at that point it sees that r2 has elected itself DR and it keeps
it that way.
This is by design of the system. With our tight ospf timers as
well as high load being experienced on our test systems. There
exists a bunch of ospf tests that we cannot guarantee that a
consistent DR will be elected for the test. As such let's not
even pretend that we care a bunch and just look for `Full`.
If we care about `ordering` we need to spend more time getting
the tests to actually start routers, ensure that htey are up and
running in the right order so that priority can take place.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Fix a loop in the setup phase of isis_topo1_vrf: only configure
interfaces that each router actually has.
Signed-off-by: Mark Stapp <mstapp@nvidia.com>
Ensure GR helpers have received a Grace-LSA before killing the
ospfd/ospf6d process that is undergoing a graceful restart.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
There's no more difference between number-named and word-named access-lists.
This commit removes separate arguments for number-named ACLs from CLI.
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
BGP LU will use implicit-null in more situations now; adjust
the original LU topotest to align with that. Node R2 uses
imp-null now, while R1 continues to allocate labels.
Signed-off-by: Mark Stapp <mstapp@nvidia.com>
