Under heavy system load we can see that the static_simple
test is giving up too early in this micronet run:
8-17 15:00:27,105 DEBUG: topo: Waiting for [0.1]s as initial delay
2023-08-17 15:00:27,206 DEBUG: r1: cmd_status("/bin/bash -c 'ip -4 route show'")
2023-08-17 15:00:28,209 DEBUG: r1:
stdout: 101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:28,209 DEBUG: topo: checking kernel routing table:
101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:28,210 INFO: topo: Function raised exception: Failed to find
'10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20'
in
'101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
'
assert None
+ where None = <function search at 0x7f405b7bb0a0>('10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20', '101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1 \n')
+ where <function search at 0x7f405b7bb0a0> = re.search
2023-08-17 15:00:28,210 DEBUG: topo: Sleeping 2s until next retry with 3.0 retry time left
2023-08-17 15:00:30,211 DEBUG: r1: cmd_status("/bin/bash -c 'ip -4 route show'")
2023-08-17 15:00:31,703 DEBUG: r1:
stdout: 101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:31,703 DEBUG: topo: checking kernel routing table:
101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:31,704 INFO: topo: Function raised exception: Failed to find
'10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20'
in
'101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
'
assert None
+ where None = <function search at 0x7f405b7bb0a0>('10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20', '101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1 \n')
+ where <function search at 0x7f405b7bb0a0> = re.search
2023-08-17 15:00:31,704 INFO: topo: Retry timeout of 3s reached
2023-08-17 15:00:31,704 INFO: topo: Spawn collection of support bundle for r1
2023-08-17 15:00:31,704 DEBUG: r1: cmd_status("/bin/bash -c 'mkdir -p /tmp/topotests/static_simple.test_static_simple/r1/support_bundles/test_static_cli'")
2023-08-17 15:00:31,710 DEBUG: r1: popen("/usr/lib/frr/generate_support_bundle.py --log-dir=/tmp/topotests/static_simple.test_static_simple/r1/support_bundles/test_static_cli")
2023-08-17 15:00:31,711 DEBUG: topo: Waiting on support bundle for r1
2023-08-17 15:00:31,751 DEBUG: topo: RETRY DIAG: [failure] Sleeping 2s until next retry with 2.2 retry time left - too see if timeout was too short
2023-08-17 15:00:33,751 DEBUG: r1: cmd_status("/bin/bash -c 'ip -4 route show'")
2023-08-17 15:00:35,137 DEBUG: r1:
stdout: 10.0.0.0/8 nhid 12 via 101.0.0.2 dev r1-eth0 proto 196 metric 20...
2023-08-17 15:00:35,137 DEBUG: topo: checking kernel routing table:
10.0.0.0/8 nhid 12 via 101.0.0.2 dev r1-eth0 proto 196 metric 20
101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
2023-08-17 15:00:35,137 DEBUG: topo: Function returned None
2023-08-17 15:00:35,138 WARN: topo: RETRY DIAGNOSTIC: SUCCEED after FAILED with requested timeout of 3.0s; however, succeeded in 8.0s, investigate timeout timing
2023-08-17 15:00:35,138 INFO: topo: Function raised exception: Failed to find
'10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20'
in
'101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1
'
assert None
+ where None = <function search at 0x7f405b7bb0a0>('10.0.0.0/8(?: nhid [0-9]+)? via 101.0.0.2 dev r1-eth0 proto (static|196) metric 20', '101.0.0.0/24 dev r1-eth0 proto kernel scope link src 101.0.0.1 \n')
+ where <function search at 0x7f405b7bb0a0> = re.search
2023-08-17 15:00:35,138 DEBUG: topo: RETRY DIAG: [failure] Sleeping 2s until next retry with 0.2 retry time left - too see if timeout was too short
2023-08-17 15:00:37,139 DEBUG: r1: cmd_status("/bin/bash -c 'ip -4 route show'")
2023-08-17 15:00:37,247 DEBUG: r1:
stdout: 10.0.0.0/8 nhid 12 via 101.0.0.2 dev r1-eth0 proto 196 metric 20...
Of course it works in the extra couple of times it tries but the test still fails.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
As part of the conversion to a `struct peer_connection` it will
be desirable to have 2 pointers one for when we open a connection
and one for when we receive a connection. Start this actual
conversion over to this in `struct peer`. If this sounds confusing
take a look at the bgp state machine for connections and how
it resolves the processing of this router opening -vs- this
router receiving an open. At some point in time the state
machine decides that we are keeping one of the two connections.
Future commits will allow us to untangle the peer/doppelganger
duality with this abstraction.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The status and ostatus are a function of the `struct peer_connection`
move it into that data structure.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Move the peer->t_write and peer->t_read into `struct peer_connection`
as that these are properties of the connection.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
P# Please enter the commit message for your changes. Lines starting
BGP tracks connections based upon the peer. But the problem
with this is that the doppelganger structure for it is being
created. This has introduced a bunch of fragileness in that
the peer exists independently of the connections to it.
The whole point of the doppelganger structure was to allow
BGP to both accept and initiate tcp connections and then
when we get one to a `good` state we collapse into the
appropriate one. The problem with this is that having
2 peer structures for this creates a situation where
we have to make sure we are configing the `right` one
and also make sure that we collapse the two independent
peer structures into 1 acting peer. This makes no sense
let's abstract out the peer into having 2 connection
one for incoming connections and one for outgoing connections
then we can easily collapse down without having to do crazy
stuff. In addition people adding new features don't need
to have to go touch a million places in the code.
This is the start of this abstraction. In this commit
we'll just pull out the fd and input/output buffers
into a connection data structure. Future commits
will abstract further.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The tests were originally tor --- spine
lets add a tor -- leaf -- spine. At this
point this change was to allow me to test
some funkiness I am seeing in pim vxlan setups
when the leaf is acting as the intermediate routers.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
There is no test that checks for the mpls interface
configuration.
The new test checks that mpls configuration per
interface works when value is enabled or disabled.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
isis_snmp.test_isis_snmp/r1/ldpd.log:2023/08/04 12:49:54 LDP: [SHWNK-NWT5S][EC 100663304] No such command on config line 8: agentx
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
babel_topo1.test_babel_topo1/r3/babeld.log:2023/08/04 12:46:55 BABELD: [SHWNK-NWT5S][EC 100663304] No such command on config line 17: redistirbute ipv6 connected
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
./config_timing.test_config_timing/r1/zebra.log:2023/08/04 12:34:29 ZEBRA: [SHWNK-NWT5S][EC 100663304] No such command on config line 7: exit-route-map
./config_timing.test_config_timing/r1/zebra.log:2023/08/04 12:34:29 ZEBRA: [SHWNK-NWT5S][EC 100663304] No such command on config line 10: exit-route-map
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt3/ospfd.log:2023/08/04 12:46:58 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 28: passive interface lo
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt5/ospfd.log:2023/08/04 12:46:59 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 27: passive interface lo
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt1/ospfd.log:2023/08/04 12:46:56 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 30: passive interface lo
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt4/ospfd.log:2023/08/04 12:47:00 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 27: passive interface lo
./bfd_ospf_topo1.test_bfd_ospf_topo1/rt2/ospfd.log:2023/08/04 12:46:57 OSPF: [SHWNK-NWT5S][EC 100663304] No such command on config line 28: passive interface lo
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The test was reading in the bgp config for the isis config and
clearly the test is working without this. So let's remove
from the test the usage of isisd
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The output of gen_json_diff_report is used all over the place and
it outputs d1 and d2. Let's change this to output and expected
as that is how it is used. Should help with debugging.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Let us start using output and expected in lib/topotest.py
because when we see output it is confusing what d1 is
versus what d2 is.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Current isis tests use a variety of hello timers as well
as hello-multiplier, let's modify all of the isis test
cases to use 1 and 10. This cleans up some spurious test
failures I was seeing locally. As an example without
these changes running isis_tilfa_topo1 2r6 times I would
see 5-10 test failures now I am seeing ~2 test failures.
In any event part of the problem was that some tests were
not fully converged when looking at them under heavy
system load. Changing this to 1/10 gives us 10 chances
to see the incoming packet.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
This test was failing upstream a bunch of times. Upon examining
the log files as well as the test script it was noticed that
the bfd peers were checked to see that they had come up. But
both the timers used for bgp as well as not checking that bgp
has actually come up would cause the test to fail in subsuquent
steps if bgp has not come up. Test that bgp peering is actually
established before testing link down events. It's possible
this test might need to be revisited to ensure that the routes
are actually installed and ready to go before as well, but I am
not seeing that right now.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
This test is failing upstream regularly, when inspecting the log
files we see that the route being looked for is in a queued state
when the test fails. Give this test more time for when the
system is under severe load.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Upstream ( and locally ) this test fails. The adj-sid value
being looked for in the testing is a dynamic value that is
assigned based upon how the network comes up. The reality
is that there is no enforced order of what the adj-sid
can be. As such this test looking for this value makes
no sense. Let's remove that from the test.
Additionally bring the isis hello-interval to 1 down
from 3 to make things converge faster.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Ran test under high load and system rejected the sharp
install of routes. Only reason that that would happen
would be if the address had not been set by the kernel
yet. The test log files had timestamp precision and the
addition of the sharp routes was under 1/10 of a second
after the address was attempted to be installed.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
