proxmox-mirrors/mirror_frr
1
0
Fork 0
mirror of https://git.proxmox.com/git/mirror_frr synced 2025-08-07 11:25:41 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

tests: new improved simple JSON template w/ doc update

Browse Source 
Utilizes new pytest fixtures to completely factor out setup and teardown
functionality. Supply the JSON config and write your tests.

"The best topotest template yet!"

Signed-off-by: Christian Hopps <chopps@labn.net>
This commit is contained in:
Christian Hopps 2021-09-05 02:24:50 -04:00
parent 5653bb515c
commit 36c1949730
No known key found for this signature in database
GPG Key ID: 2E1D830ED7B83025
4 changed files with 326 additions and 88 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

157
doc/developer/topotests-jsontopo.rst
View File

@ -23,19 +23,18 @@ On top of current topotests framework following enhancements are done:
Logging of test case executions Logging of test case executions
------------------------------- -------------------------------
* The user can enable logging of testcases execution messages into log file by * The execution log for each test is saved in the test specific directory create
adding ``frrtest_log_dir = /tmp/topotests/`` in :file:`pytest.ini`. under `/tmp/topotests` (e.g.,
* Router's current configuration can be displyed on console or sent to logs by `/tmp/topotests/<testdirname.testfilename>/exec.log`)
adding ``show_router_config = True`` in :file:`pytest.ini`.
Log file name will be displayed when we start execution: * Additionally all test logs are captured in the `topotest.xml` results file.
This file will be saved in `/tmp/topotests/topotests.xml`. In order to extract
the logs for a particular test one can use the `analyze.py` utility found in
the topotests base directory.
.. code-block:: console * Router's current configuration, as it is changed during the test, can be
displayed on console or sent to logs by adding ``show_router_config = True`` in
root@test:# python ./test_topo_json_single_link.py :file:`pytest.ini`.
Logs will be sent to logfile:
/tmp/topotests/test_topo_json_single_link_11:57:01.353797
Note: directory "/tmp/topotests/" is created by topotests by default, making Note: directory "/tmp/topotests/" is created by topotests by default, making
use of same directory to save execution logs. use of same directory to save execution logs.
@ -51,18 +50,18 @@ topology test.
This is the recommended test writing routine: This is the recommended test writing routine:
* Create a json file , which will have routers and protocol configurations * Create a json file which will have routers and protocol configurations
* Create topology from json * Write and debug the tests
* Create configuration from json
* Write the tests
* Format the new code using `black <https://github.com/psf/black>`_ * Format the new code using `black <https://github.com/psf/black>`_
* Create a Pull Request * Create a Pull Request
.. Note:: .. Note::
BGP tests MUST use generous convergence timeouts - you must ensure BGP tests MUST use generous convergence timeouts - you must ensure that any
that any test involving BGP uses a convergence timeout of at least test involving BGP uses a convergence timeout that is proportional to the
130 seconds. configured BGP timers. If the timers are not reduced from their defaults this
means 130 seconds; however, it is highly recommended that timers be reduced
from the default values unless the test requires they not be.
File Hierarchy File Hierarchy
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
@ -72,21 +71,17 @@ repository hierarchy looks like this:
.. code-block:: console .. code-block:: console
$ cd path/to/topotests $ cd frr/tests/topotests
$ find ./* $ find ./*
... ...
./example-topojson-test # the basic example test topology-1 ./example_test/
./example-topojson-test/test_example_topojson.json # input json file, having ./example_test/test_template_json.json # input json file, having topology, interfaces, bgp and other configuration
topology, interfaces, bgp and other configuration ./example_test/test_template_json.py # test script to write and execute testcases
./example-topojson-test/test_example_topojson.py # test script to write and
execute testcases
... ...
./lib # shared test/topology functions ./lib # shared test/topology functions
./lib/topojson.py # library to create topology and configurations dynamically ./lib/topojson.py # library to create topology and configurations dynamically from json file
from json file ./lib/common_config.py # library to create protocol's common configurations ex- static_routes, prefix_lists, route_maps etc.
./lib/common_config.py # library to create protocol's common configurations ex- ./lib/bgp.py # library to create and test bgp configurations
static_routes, prefix_lists, route_maps etc.
./lib/bgp.py # library to create only bgp configurations
Defining the Topology and initial configuration in JSON file Defining the Topology and initial configuration in JSON file
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -370,34 +365,32 @@ Optional keywords/options in JSON:
Building topology and configurations Building topology and configurations
"""""""""""""""""""""""""""""""""""" """"""""""""""""""""""""""""""""""""
Topology and initial configuration will be created in setup_module(). Following Topology and initial configuration as well as teardown are invoked through the
is the sample code:: use of a pytest fixture::
def setup_module(mod):
json_file = "{}/my_test_name.json".format(CWD)
tgen = Topogen(json_file, mod.__name__)
# json topo object is now available in tgen.json_topo
# Starting topology, create tmp files which are loaded to routers
# to start deamons and then start routers
start_topology(tgen)
# Creating configuration from JSON
build_config_from_json(tgen)
def teardown_module(mod):
tgen = get_topogen()
# Stop toplogy and Remove tmp files
stop_topology(tgen)
* Note: Topology will be created in setup module but routers will not be from lib import fixtures
started until we load zebra.conf and bgpd.conf to routers. For all routers
dirs will be created in /tmp/topotests/<test_folder_name>/<router_name> tgen = pytest.fixture(fixtures.tgen_json, scope="module")
zebra.conf and bgpd.conf empty files will be created and laoded to routers.
All folder and files are deleted in teardown module..
# tgen is defined above
# topo is a fixture defined in ../conftest.py and automatically available
def test_bgp_convergence(tgen, topo):
bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
assert bgp_convergence
The `fixtures.topo_json` function calls `topojson.setup_module_from_json()` to
create and return a new `topogen.Topogen()` object using the JSON config file
with the same base filename as the test (i.e., `test_file.py` ->
`test_file.json`). Additionally, the fixture calls `tgen.stop_topology()` after
all the tests have run to cleanup. The function is only invoked once per
file/module (scope="module"), but the resulting object is passed to each
function that has `tgen` as an argument.
For more info on the powerful pytest fixtures feature please see `FIXTURES`_.
.. _FIXTURES: https://docs.pytest.org/en/6.2.x/fixture.html
Creating configuration files Creating configuration files
"""""""""""""""""""""""""""" """"""""""""""""""""""""""""
@ -425,49 +418,37 @@ Writing Tests
""""""""""""" """""""""""""
Test topologies should always be bootstrapped from the Test topologies should always be bootstrapped from the
example_test/test_template_json.py, because it contains important boilerplate `example_test/test_template_json.py` when possible in order to take advantage of
code that can't be avoided, like: the most recent infrastructure support code.
imports: os, sys, pytest, and topotest/topogen.
The global variable CWD (Current Working directory): which is most likely going
to be used to reference the routers configuration file location
Example: Example:
* The topology class that inherits from Mininet Topo class; * Define a module scoped fixture to setup/teardown and supply the tests with the
`Topogen` object.
.. code-block:: python .. code-block:: python
class TemplateTopo(Topo): import pytest
def build(self, *_args, **_opts): from lib import fixtures
tgen = get_topogen(self)
# topology build code tgen = pytest.fixture(fixtures.tgen_json, scope="module")
* pytest setup_module() and teardown_module() to start the topology: * Define test functions using pytest fixtures
.. code-block:: python .. code-block:: python
def setup_module(_m): from lib import bgp
tgen = Topogen(TemplateTopo)
# Starting topology, create tmp files which are loaded to routers # tgen is defined above
# to start deamons and then start routers # topo is a global available fixture defined in ../conftest.py
start_topology(tgen, CWD) def test_bgp_convergence(tgen, topo):
"Test for BGP convergence."
def teardown_module(_m): # Don't run this test if we have any failure.
tgen = get_topogen() if tgen.routers_have_failure():
pytest.skip(tgen.errors)
# Stop toplogy and Remove tmp files
stop_topology(tgen, CWD)
* ``__main__`` initialization code (to support running the script directly)
.. code-block:: python
if **name** == '\ **main**\ ':
sys.exit(pytest.main(["-s"]))
bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
assert bgp_convergence

1
tests/topotests/example_test/test_example.py
View File

@ -36,6 +36,7 @@ def test_fail_example():
assert True, "Some Text with explaination in case of failure" assert True, "Some Text with explaination in case of failure"
@pytest.mark.xfail
def test_ls_exits_zero(): def test_ls_exits_zero():
"Tests for ls command on invalid file" "Tests for ls command on invalid file"

188
tests/topotests/example_test/test_template_json.json Normal file
View File

@ -0,0 +1,188 @@
{
"address_types": ["ipv4","ipv6"],
"ipv4base":"10.0.0.0",
"ipv4mask":30,
"ipv6base":"fd00::",
"ipv6mask":64,
"link_ip_start":{"ipv4":"10.0.0.0", "v4mask":30, "ipv6":"fd00::", "v6mask":64},
"lo_prefix":{"ipv4":"1.0.", "v4mask":32, "ipv6":"2001:DB8:F::", "v6mask":128},
"routers":{
"r1":{
"links":{
"lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
"r2":{"ipv4":"auto", "ipv6":"auto"},
"r3":{"ipv4":"auto", "ipv6":"auto"}
},
"bgp":{
"local_as":"100",
"address_family": {
"ipv4": {
"unicast": {
"neighbor": {
"r2": {
"dest_link": {
"r1": {}
}
},
"r3": {
"dest_link": {
"r1": {}
}
}
}
}
},
"ipv6": {
"unicast": {
"neighbor": {
"r2": {
"dest_link": {
"r1": {}
}
},
"r3": {
"dest_link": {
"r1": {}
}
}
}
}
}
}
}
},
"r2":{
"links":{
"lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
"r1":{"ipv4":"auto", "ipv6":"auto"},
"r3":{"ipv4":"auto", "ipv6":"auto"}
},
"bgp":{
"local_as":"100",
"address_family": {
"ipv4": {
"unicast": {
"neighbor": {
"r1": {
"dest_link": {
"r2": {}
}
},
"r3": {
"dest_link": {
"r2": {}
}
}
}
}
},
"ipv6": {
"unicast": {
"neighbor": {
"r1": {
"dest_link": {
"r2": {}
}
},
"r3": {
"dest_link": {
"r2": {}
}
}
}
}
}
}
}
},
"r3":{
"links":{
"lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
"r1":{"ipv4":"auto", "ipv6":"auto"},
"r2":{"ipv4":"auto", "ipv6":"auto"},
"r4":{"ipv4":"auto", "ipv6":"auto"}
},
"bgp":{
"local_as":"100",
"address_family": {
"ipv4": {
"unicast": {
"neighbor": {
"r1": {
"dest_link": {
"r3": {}
}
},
"r2": {
"dest_link": {
"r3": {}
}
},
"r4": {
"dest_link": {
"r3": {}
}
}
}
}
},
"ipv6": {
"unicast": {
"neighbor": {
"r1": {
"dest_link": {
"r3": {}
}
},
"r2": {
"dest_link": {
"r3": {}
}
},
"r4": {
"dest_link": {
"r3": {}
}
}
}
}
}
}
}
},
"r4":{
"links":{
"lo": {"ipv4": "auto", "ipv6": "auto", "type": "loopback"},
"r3":{"ipv4":"auto", "ipv6":"auto"}
},
"bgp":{
"local_as":"200",
"address_family": {
"ipv4": {
"unicast": {
"neighbor": {
"r3": {
"dest_link": {
"r4": {}
}
}
}
}
},
"ipv6": {
"unicast": {
"neighbor": {
"r3": {
"dest_link": {
"r4": {}
}
}
}
}
}
}
}
}
}
}

68
tests/topotests/example_test/test_template_json.py Normal file
View File

@ -0,0 +1,68 @@
#!/usr/bin/env python3
#
# September 5 2021, Christian Hopps <chopps@labn.net>
#
# Copyright (c) 2021, LabN Consulting, L.L.C.
# Copyright (c) 2017 by
# Network Device Education Foundation, Inc. ("NetDEF")
#
# Permission to use, copy, modify, and/or distribute this software
# for any purpose with or without fee is hereby granted, provided
# that the above copyright notice and this permission notice appear
# in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND NETDEF DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL NETDEF BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY
# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
# OF THIS SOFTWARE.
#
"""
<template>.py: Test <template>.
"""
import pytest
# Import topogen and topotest helpers
from lib import bgp
from lib import fixtures
# TODO: select markers based on daemons used during test
pytestmark = [
pytest.mark.bgpd,
# pytest.mark.ospfd,
# pytest.mark.ospf6d
# ...
]
# Use tgen_json fixture (invoked by use test arg of same name) to
# setup/teardown standard JSON topotest
tgen = pytest.fixture(fixtures.tgen_json, scope="module")
# tgen is defined above
# topo is a fixture defined in ../conftest.py
def test_bgp_convergence(tgen, topo):
"Test for BGP convergence."
# Don't run this test if we have any failure.
if tgen.routers_have_failure():
pytest.skip(tgen.errors)
bgp_convergence = bgp.verify_bgp_convergence(tgen, topo)
assert bgp_convergence
# Memory leak test template
def test_memory_leak(tgen):
"Run the memory leak test and report results."
if not tgen.is_memleak_enabled():
pytest.skip("Memory leak test/report is disabled")
tgen.report_memory_leaks()