proxmox-mirrors/mirror_frr
1
0
Fork 0
mirror of https://git.proxmox.com/git/mirror_frr synced 2025-08-06 10:54:47 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #6253 from opensourcerouting/fpm-extra

Browse Source 
zebra/fpm: fix shutdown and add more documentation
This commit is contained in:
Quentin Young 2020-04-21 11:28:05 -04:00 committed by GitHub
commit e15361b322
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 290 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

103
doc/developer/fpm.rst Normal file
View File

@ -0,0 +1,103 @@
FPM
===
FPM stands for Forwarding Plane Manager and it's a module for use with Zebra.
The encapsulation header for the messages exchanged with the FPM is
defined by the file :file:`fpm/fpm.h` in the frr tree. The routes
themselves are encoded in Netlink or protobuf format, with Netlink
being the default.
Netlink is standard format for encoding messages to talk with kernel space
in Linux and it is also the name of the socket type used by it.
The FPM netlink usage differs from Linux's in:
- Linux netlink sockets use datagrams in a multicast fashion, FPM uses
as a stream and it is unicast.
- FPM netlink messages might have more or less information than a normal
Linux netlink socket message (example: RTM_NEWROUTE might add an extra
route attribute to signalize VxLAN encapsulation).
Protobuf is one of a number of new serialization formats wherein the
message schema is expressed in a purpose-built language. Code for
encoding/decoding to/from the wire format is generated from the
schema. Protobuf messages can be extended easily while maintaining
backward-compatibility with older code. Protobuf has the following
advantages over Netlink:
- Code for serialization/deserialization is generated automatically. This
reduces the likelihood of bugs, allows third-party programs to be integrated
quickly, and makes it easy to add fields.
- The message format is not tied to an OS (Linux), and can be evolved
independently.
.. note::
Currently there are two FPM modules in ``zebra``:
* ``fpm``
* ``dplane_fpm_nl``
fpm
^^^
The first FPM implementation that was built using hooks in ``zebra`` route
handling functions. It uses its own netlink/protobuf encoding functions to
translate ``zebra`` route data structures into formatted binary data.
dplane_fpm_nl
^^^^^^^^^^^^^
The newer FPM implementation that was built using ``zebra``'s data plane
framework as a plugin. It only supports netlink and it shares ``zebra``'s
netlink functions to translate route event snapshots into formatted binary
data.
Protocol Specification
----------------------
FPM (in any mode) uses a TCP connection to talk with external applications.
It operates as TCP client and uses the CLI configured address/port to connect
to the FPM server (defaults to port ``2620``).
FPM frames all data with a header to help the external reader figure how
many bytes it has to read in order to read the full message (this helps
simulates datagrams like in the original netlink Linux kernel usage).
Frame header:
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------------------------------+
| Version | Message type | Message length |
+---------------+---------------+-------------------------------+
| Data... |
+---------------------------------------------------------------+
Version
^^^^^^^
Currently there is only one version, so it should be always ``1``.
Message Type
^^^^^^^^^^^^
Defines what underlining protocol we are using: netlink (``1``) or protobuf (``2``).
Message Length
^^^^^^^^^^^^^^
Amount of data in this frame in network byte order.
Data
^^^^
The netlink or protobuf message payload.

1
doc/developer/index.rst
View File

@ -11,6 +11,7 @@ FRRouting Developer's Guide
library
testing
bgpd
fpm
ospf
zebra
vtysh

160
doc/user/zebra.rst
View File

@ -736,43 +736,30 @@ these cases, the FIB needs to be maintained reliably in the fast path
as well. We refer to the component that programs the forwarding plane
(directly or indirectly) as the Forwarding Plane Manager or FPM.
The FIB push interface comprises of a TCP connection between zebra and
the FPM. The connection is initiated by zebra -- that is, the FPM acts
as the TCP server.
.. program:: configure
The relevant zebra code kicks in when zebra is configured with the
:option:`--enable-fpm` flag. Zebra periodically attempts to connect to
the well-known FPM port. Once the connection is up, zebra starts
sending messages containing routes over the socket to the FPM. Zebra
sends a complete copy of the forwarding table to the FPM, including
routes that it may have picked up from the kernel. The existing
interaction of zebra with the kernel remains unchanged -- that is, the
kernel continues to receive FIB updates as before.
:option:`--enable-fpm` flag and started with the module (``-M fpm``
or ``-M dplane_fpm_nl``).
The encapsulation header for the messages exchanged with the FPM is
defined by the file :file:`fpm/fpm.h` in the frr tree. The routes
themselves are encoded in Netlink or protobuf format, with Netlink
being the default.
.. note::
Protobuf is one of a number of new serialization formats wherein the
message schema is expressed in a purpose-built language. Code for
encoding/decoding to/from the wire format is generated from the
schema. Protobuf messages can be extended easily while maintaining
backward-compatibility with older code. Protobuf has the following
advantages over Netlink:
The ``fpm`` implementation attempts to connect to ``127.0.0.1`` port ``2620``
by default without configurations. The ``dplane_fpm_nl`` only attempts to
connect to a server if configured.
- Code for serialization/deserialization is generated automatically. This
reduces the likelihood of bugs, allows third-party programs to be integrated
quickly, and makes it easy to add fields.
- The message format is not tied to an OS (Linux), and can be evolved
independently.
Zebra periodically attempts to connect to the well-known FPM port (``2620``).
Once the connection is up, zebra starts sending messages containing routes
over the socket to the FPM. Zebra sends a complete copy of the forwarding
table to the FPM, including routes that it may have picked up from the kernel.
The existing interaction of zebra with the kernel remains unchanged -- that
is, the kernel continues to receive FIB updates as before.
As mentioned before, zebra encodes routes sent to the FPM in Netlink
format by default. The format can be controlled via the FPM module's
load-time option to zebra, which currently takes the values `Netlink`
and `protobuf`.
The default FPM message format is netlink, however it can be controlled
with the module load-time option. The modules accept the following options:
- ``fpm``: ``netlink`` and ``protobuf``.
- ``dplane_fpm_nl``: none, it only implements netlink.
The zebra FPM interface uses replace semantics. That is, if a 'route
add' message for a prefix is followed by another 'route add' message,
@ -782,6 +769,119 @@ replaces the information sent in the first message.
If the connection to the FPM goes down for some reason, zebra sends
the FPM a complete copy of the forwarding table(s) when it reconnects.
For more details on the implementation, please read the developer's manual FPM
section.
FPM Commands
============
``fpm`` implementation
----------------------
.. index:: fpm connection ip A.B.C.D port (1-65535)
.. clicmd:: fpm connection ip A.B.C.D port (1-65535)
Configure ``zebra`` to connect to a different FPM server than
``127.0.0.1`` port ``2620``.
.. index:: no fpm connection ip A.B.C.D port (1-65535)
.. clicmd:: no fpm connection ip A.B.C.D port (1-65535)
Configure ``zebra`` to connect to the default FPM server at ``127.0.0.1``
port ``2620``.
.. index:: show zebra fpm stats
.. clicmd:: show zebra fpm stats
Shows the FPM statistics.
Sample output:
::
Counter Total Last 10 secs
connect_calls 3 2
connect_no_sock 0 0
read_cb_calls 2 2
write_cb_calls 2 0
write_calls 1 0
partial_writes 0 0
max_writes_hit 0 0
t_write_yields 0 0
nop_deletes_skipped 6 0
route_adds 5 0
route_dels 0 0
updates_triggered 11 0
redundant_triggers 0 0
dests_del_after_update 0 0
t_conn_down_starts 0 0
t_conn_down_dests_processed 0 0
t_conn_down_yields 0 0
t_conn_down_finishes 0 0
t_conn_up_starts 1 0
t_conn_up_dests_processed 11 0
t_conn_up_yields 0 0
t_conn_up_aborts 0 0
t_conn_up_finishes 1 0
.. index:: clear zebra fpm stats
.. clicmd:: clear zebra fpm stats
Resets all FPM counters.
``dplane_fpm_nl`` implementation
--------------------------------
.. index:: fpm address <A.B.C.D|X:X::X:X> [port (1-65535)]
.. clicmd:: fpm address <A.B.C.D|X:X::X:X> [port (1-65535)]
Configures the FPM server address. Once configured ``zebra`` will attempt
to connect to it immediately.
.. index:: no fpm address [<A.B.C.D|X:X::X:X> [port (1-65535)]]
.. clicmd:: no fpm address [<A.B.C.D|X:X::X:X> [port (1-65535)]]
Disables FPM entirely. ``zebra`` will close any current connections and
will not attempt to connect to it anymore.
.. index:: show fpm counters [json]
.. clicmd:: show fpm counters [json]
Show the FPM statistics (plain text or JSON formatted).
Sample output:
::
FPM counters
============
Input bytes: 0
Output bytes: 308
Output buffer current size: 0
Output buffer peak size: 308
Connection closes: 0
Connection errors: 0
Data plane items processed: 0
Data plane items enqueued: 0
Data plane items queue peak: 0
Buffer full hits: 0
User FPM configurations: 1
User FPM disable requests: 0
.. index:: clear fpm counters
.. clicmd:: clear fpm counters
Resets all FPM counters.
.. _zebra-dplane:
Dataplane Commands

42
zebra/dplane_fpm_nl.c
View File

@ -1027,16 +1027,50 @@ static int fpm_nl_start(struct zebra_dplane_provider *prov)
return 0;
}
static int fpm_nl_finish_early(struct fpm_nl_ctx *fnc)
{
/* Disable all events and close socket. */
THREAD_OFF(fnc->t_ribreset);
THREAD_OFF(fnc->t_ribwalk);
THREAD_OFF(fnc->t_rmacreset);
THREAD_OFF(fnc->t_rmacwalk);
thread_cancel_async(fnc->fthread->master, &fnc->t_read, NULL);
thread_cancel_async(fnc->fthread->master, &fnc->t_write, NULL);
thread_cancel_async(fnc->fthread->master, &fnc->t_connect, NULL);
if (fnc->socket != -1) {
close(fnc->socket);
fnc->socket = -1;
}
return 0;
}
static int fpm_nl_finish_late(struct fpm_nl_ctx *fnc)
{
/* Stop the running thread. */
frr_pthread_stop(fnc->fthread, NULL);
/* Free all allocated resources. */
pthread_mutex_destroy(&fnc->obuf_mutex);
pthread_mutex_destroy(&fnc->ctxqueue_mutex);
stream_free(fnc->ibuf);
stream_free(fnc->obuf);
free(gfnc);
gfnc = NULL;
return 0;
}
static int fpm_nl_finish(struct zebra_dplane_provider *prov, bool early)
{
struct fpm_nl_ctx *fnc;
fnc = dplane_provider_get_data(prov);
stream_free(fnc->ibuf);
stream_free(fnc->obuf);
close(fnc->socket);
if (early)
return fpm_nl_finish_early(fnc);
return 0;
return fpm_nl_finish_late(fnc);
}
static int fpm_nl_process(struct zebra_dplane_provider *prov)

20
zebra/zebra_dplane.c
View File

@ -3557,12 +3557,20 @@ bool dplane_is_in_shutdown(void)
*/
void zebra_dplane_pre_finish(void)
{
struct zebra_dplane_provider *dp;
if (IS_ZEBRA_DEBUG_DPLANE)
zlog_debug("Zebra dataplane pre-fini called");
zdplane_info.dg_is_shutdown = true;
/* TODO -- Notify provider(s) of pending shutdown */
/* Notify provider(s) of pending shutdown. */
TAILQ_FOREACH(dp, &zdplane_info.dg_providers_q, dp_prov_link) {
if (dp->dp_fini == NULL)
continue;
dp->dp_fini(dp, true);
}
}
/*
@ -3863,6 +3871,8 @@ done:
*/
void zebra_dplane_shutdown(void)
{
struct zebra_dplane_provider *dp;
if (IS_ZEBRA_DEBUG_DPLANE)
zlog_debug("Zebra dataplane shutdown called");
@ -3881,7 +3891,13 @@ void zebra_dplane_shutdown(void)
zdplane_info.dg_pthread = NULL;
zdplane_info.dg_master = NULL;
/* TODO -- Notify provider(s) of final shutdown */
/* Notify provider(s) of final shutdown. */
TAILQ_FOREACH(dp, &zdplane_info.dg_providers_q, dp_prov_link) {
if (dp->dp_fini == NULL)
continue;
dp->dp_fini(dp, false);
}
/* TODO -- Clean-up provider objects */