Under heavy system load we can see that the static_simple
test is giving up too early in this micronet run:
8-17 15:00:27,105 DEBUG: topo: Waiting for [0.1]s as initial delay
2023-08-17 15:00:27,206 DEBUG: r1: cmd_status("/bin/bash -c 'ip -4 route show'")
2023-08-17 15:00:28,209 DEBUG: r1:
stdout: 101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:28,209 DEBUG: topo: checking kernel routing table:
101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:28,210 INFO: topo: Function raised exception: Failed to find
'10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20'
in
'101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
'
assert None
+ where None = <function search at 0x7f405b7bb0a0>('10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20', '101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1 \n')
+ where <function search at 0x7f405b7bb0a0> = re.search
2023-08-17 15:00:28,210 DEBUG: topo: Sleeping 2s until next retry with 3.0 retry time left
2023-08-17 15:00:30,211 DEBUG: r1: cmd_status("/bin/bash -c 'ip -4 route show'")
2023-08-17 15:00:31,703 DEBUG: r1:
stdout: 101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:31,703 DEBUG: topo: checking kernel routing table:
101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:31,704 INFO: topo: Function raised exception: Failed to find
'10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20'
in
'101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
'
assert None
+ where None = <function search at 0x7f405b7bb0a0>('10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20', '101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1 \n')
+ where <function search at 0x7f405b7bb0a0> = re.search
2023-08-17 15:00:31,704 INFO: topo: Retry timeout of 3s reached
2023-08-17 15:00:31,704 INFO: topo: Spawn collection of support bundle for r1
2023-08-17 15:00:31,704 DEBUG: r1: cmd_status("/bin/bash -c 'mkdir -p /tmp/topotests/static_simple.test_static_simple/r1/support_bundles/test_static_cli'")
2023-08-17 15:00:31,710 DEBUG: r1: popen("/usr/lib/frr/generate_support_bundle.py --log-dir=/tmp/topotests/static_simple.test_static_simple/r1/support_bundles/test_static_cli")
2023-08-17 15:00:31,711 DEBUG: topo: Waiting on support bundle for r1
2023-08-17 15:00:31,751 DEBUG: topo: RETRY DIAG: [failure] Sleeping 2s until next retry with 2.2 retry time left - too see if timeout was too short
2023-08-17 15:00:33,751 DEBUG: r1: cmd_status("/bin/bash -c 'ip -4 route show'")
2023-08-17 15:00:35,137 DEBUG: r1:
stdout: 10.0.0.0/8 nhid 12 via 101.0.0.2 dev r1-eth0 proto 196 metric 20...
2023-08-17 15:00:35,137 DEBUG: topo: checking kernel routing table:
10.0.0.0/8 nhid 12 via 101.0.0.2 dev r1-eth0 proto 196 metric 20
101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:35,137 DEBUG: topo: Function returned None
2023-08-17 15:00:35,138 WARN: topo: RETRY DIAGNOSTIC: SUCCEED after FAILED with requested timeout of 3.0s; however, succeeded in 8.0s, investigate timeout timing
2023-08-17 15:00:35,138 INFO: topo: Function raised exception: Failed to find
'10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20'
in
'101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
'
assert None
+ where None = <function search at 0x7f405b7bb0a0>('10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20', '101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1 \n')
+ where <function search at 0x7f405b7bb0a0> = re.search
2023-08-17 15:00:35,138 DEBUG: topo: RETRY DIAG: [failure] Sleeping 2s until next retry with 0.2 retry time left - too see if timeout was too short
2023-08-17 15:00:37,139 DEBUG: r1: cmd_status("/bin/bash -c 'ip -4 route show'")
2023-08-17 15:00:37,247 DEBUG: r1:
stdout: 10.0.0.0/8 nhid 12 via 101.0.0.2 dev r1-eth0 proto 196 metric 20...
Of course it works in the extra couple of times it tries but the test still fails.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The tests were originally tor --- spine
lets add a tor -- leaf -- spine. At this
point this change was to allow me to test
some funkiness I am seeing in pim vxlan setups
when the leaf is acting as the intermediate routers.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
There is no test that checks for the mpls interface
configuration.
The new test checks that mpls configuration per
interface works when value is enabled or disabled.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
isis_snmp.test_isis_snmp/r1/ldpd.log:2023/08/04 12:49:54 LDP: [SHWNK-NWT5S][EC 100663304] No such command on config line 8: agentx
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
babel_topo1.test_babel_topo1/r3/babeld.log:2023/08/04 12:46:55 BABELD: [SHWNK-NWT5S][EC 100663304] No such command on config line 17: redistirbute ipv6 connected
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
./config_timing.test_config_timing/r1/zebra.log:2023/08/04 12:34:29 ZEBRA: [SHWNK-NWT5S][EC 100663304] No such command on config line 7: exit-route-map
./config_timing.test_config_timing/r1/zebra.log:2023/08/04 12:34:29 ZEBRA: [SHWNK-NWT5S][EC 100663304] No such command on config line 10: exit-route-map
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt3/ospfd.log:2023/08/04 12:46:58 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 28: passive interface lo
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt5/ospfd.log:2023/08/04 12:46:59 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 27: passive interface lo
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt1/ospfd.log:2023/08/04 12:46:56 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 30: passive interface lo
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt4/ospfd.log:2023/08/04 12:47:00 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 27: passive interface lo
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt2/ospfd.log:2023/08/04 12:46:57 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 28: passive interface lo
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The test was reading in the bgp config for the isis config and
clearly the test is working without this. So let's remove
from the test the usage of isisd
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The output of gen_json_diff_report is used all over the place and
it outputs d1 and d2. Let's change this to output and expected
as that is how it is used. Should help with debugging.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Let us start using output and expected in lib/topotest.py
because when we see output it is confusing what d1 is
versus what d2 is.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Current isis tests use a variety of hello timers as well
as hello-multiplier, let's modify all of the isis test
cases to use 1 and 10. This cleans up some spurious test
failures I was seeing locally. As an example without
these changes running isis_tilfa_topo1 2r6 times I would
see 5-10 test failures now I am seeing ~2 test failures.
In any event part of the problem was that some tests were
not fully converged when looking at them under heavy
system load. Changing this to 1/10 gives us 10 chances
to see the incoming packet.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
This test was failing upstream a bunch of times. Upon examining
the log files as well as the test script it was noticed that
the bfd peers were checked to see that they had come up. But
both the timers used for bgp as well as not checking that bgp
has actually come up would cause the test to fail in subsuquent
steps if bgp has not come up. Test that bgp peering is actually
established before testing link down events. It's possible
this test might need to be revisited to ensure that the routes
are actually installed and ready to go before as well, but I am
not seeing that right now.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
This test is failing upstream regularly, when inspecting the log
files we see that the route being looked for is in a queued state
when the test fails. Give this test more time for when the
system is under severe load.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Upstream ( and locally ) this test fails. The adj-sid value
being looked for in the testing is a dynamic value that is
assigned based upon how the network comes up. The reality
is that there is no enforced order of what the adj-sid
can be. As such this test looking for this value makes
no sense. Let's remove that from the test.
Additionally bring the isis hello-interval to 1 down
from 3 to make things converge faster.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Ran test under high load and system rejected the sharp
install of routes. Only reason that that would happen
would be if the address had not been set by the kernel
yet. The test log files had timestamp precision and the
addition of the sharp routes was under 1/10 of a second
after the address was attempted to be installed.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Starting from step 11, this topotest focuses on validating the TI-LFA
switchover functionality, where the backup nexthops are activated
after an adjacency expires, either with or without BFD.
Currently, the test checks the RIB shortly after the switchover using
a tight 5 seconds interval to ensure that the RIB update is due to the
switchover and not an SPF update (which is configured with an initial
delay of 15 seconds). However, it was observed that the kernel might
take longer than 5 seconds to install routes when the system is under
heavy load. To account for that, double the wait interval so that
this topotest will succeed even in those conditions.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
In this topotest, BFD is configured at the end of step 13. However,
in certain cases where the testing machine is exceptionally fast (e.g.,
Donald's quantum computer), there is a possibility that the interface
shutdown event from step 14 may occur before BFD has had sufficient
time to establish the session, which leads to a test failure. To fix
this problem, ensure the BFD session is up before proceeding to the
next step.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
In this topotest, the IS-IS hello interval is set to 1 for fast
convergence. However, the current hello multiplier of 3 results in a
tight IS-IS adjacency holdtime of 3 seconds. This tight timeframe can
cause failures when the testing machine is running multiple tests at
full capacity. To improve stability under such conditions, this commit
raises the hello multiplier to 10, providing a more forgiving holdtime
and reducing the likelihood of failures.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
bgpd, pbrd: use common pbr encoder
zebra: use common pbr decoder
tests: pbr_topo1: check more filter fields
Purpose:
1. Reduce likelihood of zapi format mismatches when adding
PBR fields due to multiple parallel encoder implementations
2. Encourage common PBR structure usage among various daemons
3. Reduce coding errors via explicit per-field enable flags
Signed-off-by: G. Paul Ziemba <paulz@labn.net>
