mirror of
https://git.proxmox.com/git/mirror_frr
synced 2025-08-03 11:01:48 +00:00
tests: new improved simple JSON template w/ doc update
Utilizes new pytest fixtures to completely factor out setup and teardown functionality. Supply the JSON config and write your tests. "The best topotest template yet!" Signed-off-by: Christian Hopps <chopps@labn.net>
This commit is contained in:
Christian Hopps
parent
5653bb515c
commit
36c1949730
@ -23,19 +23,18 @@ On top of current topotests framework following enhancements are done:
|
||||
Logging of test case executions
|
||||
-------------------------------
|
||||
|
||||
* The user can enable logging of testcases execution messages into log file by
|
||||
adding ``frrtest_log_dir = /tmp/topotests/`` in :file:`pytest.ini`.
|
||||
* Router's current configuration can be displyed on console or sent to logs by
|
||||
adding ``show_router_config = True`` in :file:`pytest.ini`.
|
||||
* The execution log for each test is saved in the test specific directory create
|
||||
under `/tmp/topotests` (e.g.,
|
||||
`/tmp/topotests/<testdirname.testfilename>/exec.log`)
|
||||
|
||||
Log file name will be displayed when we start execution:
|
||||
* Additionally all test logs are captured in the `topotest.xml` results file.
|
||||
This file will be saved in `/tmp/topotests/topotests.xml`. In order to extract
|
||||
the logs for a particular test one can use the `analyze.py` utility found in
|
||||
the topotests base directory.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
root@test:# python ./test_topo_json_single_link.py
|
||||
|
||||
Logs will be sent to logfile:
|
||||
/tmp/topotests/test_topo_json_single_link_11:57:01.353797
|
||||
* Router's current configuration, as it is changed during the test, can be
|
||||
displayed on console or sent to logs by adding ``show_router_config = True`` in
|
||||
:file:`pytest.ini`.
|
||||
|
||||
Note: directory "/tmp/topotests/" is created by topotests by default, making
|
||||
use of same directory to save execution logs.
|
||||
@ -51,18 +50,18 @@ topology test.
|
||||
|
||||
This is the recommended test writing routine:
|
||||
|
||||
* Create a json file , which will have routers and protocol configurations
|
||||
* Create topology from json
|
||||
* Create configuration from json
|
||||
* Write the tests
|
||||
* Create a json file which will have routers and protocol configurations
|
||||
* Write and debug the tests
|
||||
* Format the new code using `black <https://github.com/psf/black>`_
|
||||
* Create a Pull Request
|
||||
|
||||
.. Note::
|
||||
|
||||
BGP tests MUST use generous convergence timeouts - you must ensure
|
||||
that any test involving BGP uses a convergence timeout of at least
|
||||
130 seconds.
|
||||
BGP tests MUST use generous convergence timeouts - you must ensure that any
|
||||
test involving BGP uses a convergence timeout that is proportional to the
|
||||
configured BGP timers. If the timers are not reduced from their defaults this
|
||||
means 130 seconds; however, it is highly recommended that timers be reduced
|
||||
from the default values unless the test requires they not be.
|
||||
|
||||
File Hierarchy
|
||||
^^^^^^^^^^^^^^
|
||||
@ -72,21 +71,17 @@ repository hierarchy looks like this:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cd path/to/topotests
|
||||
$ cd frr/tests/topotests
|
||||
$ find ./*
|
||||
...
|
||||
./example-topojson-test # the basic example test topology-1
|
||||
./example-topojson-test/test_example_topojson.json # input json file, having
|
||||
topology, interfaces, bgp and other configuration
|
||||
./example-topojson-test/test_example_topojson.py # test script to write and
|
||||
execute testcases
|
||||
./example_test/
|
||||
./example_test/test_template_json.json # input json file, having topology, interfaces, bgp and other configuration
|
||||
./example_test/test_template_json.py # test script to write and execute testcases
|
||||
...
|
||||
./lib # shared test/topology functions
|
||||
./lib/topojson.py # library to create topology and configurations dynamically
|
||||
from json file
|
||||
./lib/common_config.py # library to create protocol's common configurations ex-
|
||||
static_routes, prefix_lists, route_maps etc.
|
||||
./lib/bgp.py # library to create only bgp configurations
|
||||
./lib/topojson.py # library to create topology and configurations dynamically from json file
|
||||
./lib/common_config.py # library to create protocol's common configurations ex- static_routes, prefix_lists, route_maps etc.
|
||||
./lib/bgp.py # library to create and test bgp configurations
|
||||
|
||||
Defining the Topology and initial configuration in JSON file
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -370,34 +365,32 @@ Optional keywords/options in JSON:
|
||||
Building topology and configurations
|
||||
""""""""""""""""""""""""""""""""""""
|
||||
|
||||
Topology and initial configuration will be created in setup_module(). Following
|
||||
is the sample code::
|
||||
|
||||
def setup_module(mod):
|
||||
json_file = "{}/my_test_name.json".format(CWD)
|
||||
tgen = Topogen(json_file, mod.__name__)
|
||||
|
||||
# json topo object is now available in tgen.json_topo
|
||||
|
||||
# Starting topology, create tmp files which are loaded to routers
|
||||
# to start deamons and then start routers
|
||||
start_topology(tgen)
|
||||
|
||||
# Creating configuration from JSON
|
||||
build_config_from_json(tgen)
|
||||
|
||||
def teardown_module(mod):
|
||||
tgen = get_topogen()
|
||||
|
||||
# Stop toplogy and Remove tmp files
|
||||
stop_topology(tgen)
|
||||
Topology and initial configuration as well as teardown are invoked through the
|
||||
use of a pytest fixture::
|
||||
|
||||
|
||||
* Note: Topology will be created in setup module but routers will not be
|
||||
started until we load zebra.conf and bgpd.conf to routers. For all routers
|
||||
dirs will be created in /tmp/topotests/<test_folder_name>/<router_name>
|
||||
zebra.conf and bgpd.conf empty files will be created and laoded to routers.
|
||||
All folder and files are deleted in teardown module..
|
||||
from lib import fixtures
|
||||
|
||||
tgen = pytest.fixture(fixtures.tgen_json, scope="module")
|
||||
|
||||
|
||||
# tgen is defined above
|
||||
# topo is a fixture defined in ../conftest.py and automatically available
|
||||
def test_bgp_convergence(tgen, topo):
|
||||
bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
|
||||
assert bgp_convergence
|
||||
|
||||
The `fixtures.topo_json` function calls `topojson.setup_module_from_json()` to
|
||||
create and return a new `topogen.Topogen()` object using the JSON config file
|
||||
with the same base filename as the test (i.e., `test_file.py` ->
|
||||
`test_file.json`). Additionally, the fixture calls `tgen.stop_topology()` after
|
||||
all the tests have run to cleanup. The function is only invoked once per
|
||||
file/module (scope="module"), but the resulting object is passed to each
|
||||
function that has `tgen` as an argument.
|
||||
|
||||
For more info on the powerful pytest fixtures feature please see `FIXTURES`_.
|
||||
|
||||
.. _FIXTURES: https://docs.pytest.org/en/6.2.x/fixture.html
|
||||
|
||||
Creating configuration files
|
||||
""""""""""""""""""""""""""""
|
||||
@ -425,49 +418,37 @@ Writing Tests
|
||||
"""""""""""""
|
||||
|
||||
Test topologies should always be bootstrapped from the
|
||||
example_test/test_template_json.py, because it contains important boilerplate
|
||||
code that can't be avoided, like:
|
||||
|
||||
imports: os, sys, pytest, and topotest/topogen.
|
||||
|
||||
The global variable CWD (Current Working directory): which is most likely going
|
||||
to be used to reference the routers configuration file location
|
||||
`example_test/test_template_json.py` when possible in order to take advantage of
|
||||
the most recent infrastructure support code.
|
||||
|
||||
Example:
|
||||
|
||||
|
||||
* The topology class that inherits from Mininet Topo class;
|
||||
* Define a module scoped fixture to setup/teardown and supply the tests with the
|
||||
`Topogen` object.
|
||||
|
||||
.. code-block:: python
|
||||
.. code-block:: python
|
||||
|
||||
class TemplateTopo(Topo):
|
||||
def build(self, *_args, **_opts):
|
||||
tgen = get_topogen(self)
|
||||
# topology build code
|
||||
import pytest
|
||||
from lib import fixtures
|
||||
|
||||
tgen = pytest.fixture(fixtures.tgen_json, scope="module")
|
||||
|
||||
|
||||
* pytest setup_module() and teardown_module() to start the topology:
|
||||
* Define test functions using pytest fixtures
|
||||
|
||||
.. code-block:: python
|
||||
.. code-block:: python
|
||||
|
||||
def setup_module(_m):
|
||||
tgen = Topogen(TemplateTopo)
|
||||
from lib import bgp
|
||||
|
||||
# Starting topology, create tmp files which are loaded to routers
|
||||
# to start deamons and then start routers
|
||||
start_topology(tgen, CWD)
|
||||
# tgen is defined above
|
||||
# topo is a global available fixture defined in ../conftest.py
|
||||
def test_bgp_convergence(tgen, topo):
|
||||
"Test for BGP convergence."
|
||||
|
||||
def teardown_module(_m):
|
||||
tgen = get_topogen()
|
||||
|
||||
# Stop toplogy and Remove tmp files
|
||||
stop_topology(tgen, CWD)
|
||||
|
||||
|
||||
* ``__main__`` initialization code (to support running the script directly)
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
if **name** == '\ **main**\ ':
|
||||
sys.exit(pytest.main(["-s"]))
|
||||
# Don't run this test if we have any failure.
|
||||
if tgen.routers_have_failure():
|
||||
pytest.skip(tgen.errors)
|
||||
|
||||
bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
|
||||
assert bgp_convergence
|
||||
|
||||
@ -36,6 +36,7 @@ def test_fail_example():
|
||||
assert True, "Some Text with explaination in case of failure"
|
||||
|
||||
|
||||
@pytest.mark.xfail
|
||||
def test_ls_exits_zero():
|
||||
"Tests for ls command on invalid file"
|
||||
|
||||
|
||||
188
tests/topotests/example_test/test_template_json.json
Normal file
188
tests/topotests/example_test/test_template_json.json
Normal file
@ -0,0 +1,188 @@
|
||||
|
||||
{
|
||||
"address_types": ["ipv4","ipv6"],
|
||||
"ipv4base":"10.0.0.0",
|
||||
"ipv4mask":30,
|
||||
"ipv6base":"fd00::",
|
||||
"ipv6mask":64,
|
||||
"link_ip_start":{"ipv4":"10.0.0.0", "v4mask":30, "ipv6":"fd00::", "v6mask":64},
|
||||
"lo_prefix":{"ipv4":"1.0.", "v4mask":32, "ipv6":"2001:DB8:F::", "v6mask":128},
|
||||
"routers":{
|
||||
"r1":{
|
||||
"links":{
|
||||
"lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
|
||||
"r2":{"ipv4":"auto", "ipv6":"auto"},
|
||||
"r3":{"ipv4":"auto", "ipv6":"auto"}
|
||||
},
|
||||
"bgp":{
|
||||
"local_as":"100",
|
||||
"address_family": {
|
||||
"ipv4": {
|
||||
"unicast": {
|
||||
"neighbor": {
|
||||
"r2": {
|
||||
"dest_link": {
|
||||
"r1": {}
|
||||
}
|
||||
},
|
||||
"r3": {
|
||||
"dest_link": {
|
||||
"r1": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ipv6": {
|
||||
"unicast": {
|
||||
"neighbor": {
|
||||
"r2": {
|
||||
"dest_link": {
|
||||
"r1": {}
|
||||
}
|
||||
},
|
||||
"r3": {
|
||||
"dest_link": {
|
||||
"r1": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"r2":{
|
||||
"links":{
|
||||
"lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
|
||||
"r1":{"ipv4":"auto", "ipv6":"auto"},
|
||||
"r3":{"ipv4":"auto", "ipv6":"auto"}
|
||||
},
|
||||
"bgp":{
|
||||
"local_as":"100",
|
||||
"address_family": {
|
||||
"ipv4": {
|
||||
"unicast": {
|
||||
"neighbor": {
|
||||
"r1": {
|
||||
"dest_link": {
|
||||
"r2": {}
|
||||
}
|
||||
},
|
||||
"r3": {
|
||||
"dest_link": {
|
||||
"r2": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ipv6": {
|
||||
"unicast": {
|
||||
"neighbor": {
|
||||
"r1": {
|
||||
"dest_link": {
|
||||
"r2": {}
|
||||
}
|
||||
},
|
||||
"r3": {
|
||||
"dest_link": {
|
||||
"r2": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"r3":{
|
||||
"links":{
|
||||
"lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
|
||||
"r1":{"ipv4":"auto", "ipv6":"auto"},
|
||||
"r2":{"ipv4":"auto", "ipv6":"auto"},
|
||||
"r4":{"ipv4":"auto", "ipv6":"auto"}
|
||||
},
|
||||
"bgp":{
|
||||
"local_as":"100",
|
||||
"address_family": {
|
||||
"ipv4": {
|
||||
"unicast": {
|
||||
"neighbor": {
|
||||
"r1": {
|
||||
"dest_link": {
|
||||
"r3": {}
|
||||
}
|
||||
},
|
||||
"r2": {
|
||||
"dest_link": {
|
||||
"r3": {}
|
||||
}
|
||||
},
|
||||
"r4": {
|
||||
"dest_link": {
|
||||
"r3": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ipv6": {
|
||||
"unicast": {
|
||||
"neighbor": {
|
||||
"r1": {
|
||||
"dest_link": {
|
||||
"r3": {}
|
||||
}
|
||||
},
|
||||
"r2": {
|
||||
"dest_link": {
|
||||
"r3": {}
|
||||
}
|
||||
},
|
||||
"r4": {
|
||||
"dest_link": {
|
||||
"r3": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"r4":{
|
||||
"links":{
|
||||
"lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
|
||||
"r3":{"ipv4":"auto", "ipv6":"auto"}
|
||||
},
|
||||
"bgp":{
|
||||
"local_as":"200",
|
||||
"address_family": {
|
||||
"ipv4": {
|
||||
"unicast": {
|
||||
"neighbor": {
|
||||
"r3": {
|
||||
"dest_link": {
|
||||
"r4": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ipv6": {
|
||||
"unicast": {
|
||||
"neighbor": {
|
||||
"r3": {
|
||||
"dest_link": {
|
||||
"r4": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
68
tests/topotests/example_test/test_template_json.py
Normal file
68
tests/topotests/example_test/test_template_json.py
Normal file
@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env python3
|
||||
#
|
||||
# September 5 2021, Christian Hopps <chopps@labn.net>
|
||||
#
|
||||
# Copyright (c) 2021, LabN Consulting, L.L.C.
|
||||
# Copyright (c) 2017 by
|
||||
# Network Device Education Foundation, Inc. ("NetDEF")
|
||||
#
|
||||
# Permission to use, copy, modify, and/or distribute this software
|
||||
# for any purpose with or without fee is hereby granted, provided
|
||||
# that the above copyright notice and this permission notice appear
|
||||
# in all copies.
|
||||
#
|
||||
# THE SOFTWARE IS PROVIDED "AS IS" AND NETDEF DISCLAIMS ALL WARRANTIES
|
||||
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
|
||||
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL NETDEF BE LIABLE FOR
|
||||
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY
|
||||
# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
|
||||
# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
|
||||
# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
|
||||
# OF THIS SOFTWARE.
|
||||
#
|
||||
|
||||
"""
|
||||
<template>.py: Test <template>.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
# Import topogen and topotest helpers
|
||||
from lib import bgp
|
||||
from lib import fixtures
|
||||
|
||||
|
||||
# TODO: select markers based on daemons used during test
|
||||
pytestmark = [
|
||||
pytest.mark.bgpd,
|
||||
# pytest.mark.ospfd,
|
||||
# pytest.mark.ospf6d
|
||||
# ...
|
||||
]
|
||||
|
||||
# Use tgen_json fixture (invoked by use test arg of same name) to
|
||||
# setup/teardown standard JSON topotest
|
||||
tgen = pytest.fixture(fixtures.tgen_json, scope="module")
|
||||
|
||||
|
||||
# tgen is defined above
|
||||
# topo is a fixture defined in ../conftest.py
|
||||
def test_bgp_convergence(tgen, topo):
|
||||
"Test for BGP convergence."
|
||||
|
||||
# Don't run this test if we have any failure.
|
||||
if tgen.routers_have_failure():
|
||||
pytest.skip(tgen.errors)
|
||||
|
||||
bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
|
||||
assert bgp_convergence
|
||||
|
||||
|
||||
# Memory leak test template
|
||||
def test_memory_leak(tgen):
|
||||
"Run the memory leak test and report results."
|
||||
|
||||
if not tgen.is_memleak_enabled():
|
||||
pytest.skip("Memory leak test/report is disabled")
|
||||
|
||||
tgen.report_memory_leaks()
|
||||
Loading…
Reference in New Issue
Block a user