Make it possible to load YANG modules outside the main northbound
initialization. The primary use case is to support YANG modules
that are specific to an FRR plugin. Example: only load the PCEP
YANG module when the corresponding FRR plugin is loaded. Other use
cases might arise in the future.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Combine yang_snodes_iterate_module() and yang_snodes_iterate_all()
into an unified yang_snodes_iterate() function, where the first
"module" parameter is optional. There's no point in having two
separate YANG schema iteration functions anymore now that they are
too similar.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
The only safe way to iterate over all schema nodes of a given YANG
module is by iterating over all schema nodes of all YANG modules
and filter out the nodes that belong to other modules.
The original yang_snodes_iterate_module() code did the following:
1 - Iterate over all top-level schema nodes of the given module;
2 - Iterate over all augmentations of the given module.
While that iteration strategy is more efficient, it does't handle
well more complex YANG hierarchies containing nested augmentations
or self-augmenting modules. Any iteration that isn't done on the
resolved YANG data hierarchy is fragile and prone to errors.
Fixes regression introduced by commit 8a923b4851 where the
gen_northbound_callbacks tool was generating duplicate callbacks
for certain modules.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Display human readable error message in northbound rpc
transaction failure. In case of vtysh nb client, the error
message will be displayed to user.
Testing:
bharat# clear evpn dup-addr vni 1002 ip 11.11.11.11
Error type: generic error
Error description: Requested IP's associated MAC aa:aa:aa:aa:aa:aa is still
in duplicate state
Signed-off-by: Chirag Shah <chirag@nvidia.com>
When calling yang_snodes_iterate_subtree we don't care about
the return code. So explicitly say we don't care so that
SA tools can be on the same page as us.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Change the way the YANG schema node iteration functions work so that
the northbound layer won't have issues with more complex YANG modules
that contain multiple levels of YANG augmentations or modules that
augment themselves indirectly (by augmenting groupings).
Summary of the changes:
* Change the yang_snodes_iterate_subtree() function to always follow
augmentations and add an optional "module" parameter to narrow down
the iteration to nodes of a single module (which is necessary in
some cases). Also, remove the YANG_ITER_ALLOW_AUGMENTATIONS flag
as it's no longer necessary.
* Change yang_snodes_iterate_all() to do a DFS iteration on the resolved
YANG data hierarchy instead of iterating over each module and their
augmentations sequentially.
Reported-by: Rafael Zalamena <rzalamena@opensourcerouting.org>
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Whenever libyang loads a module that contains a leafref, it will
also implicitly load the module of the referring node if it's
not loaded already. That makes sense as otherwise it wouldn't be
possible to validate the leafref value correctly.
The problem is that loading a module implicitly violates the
assumption of the northbound layer that all loaded modules
are implemented (i.e. they have a northbound node associated
to each schema node). This means that loading a module that
isn't implemented can lead to crashes as the "priv" pointer
of schema nodes is no longer guaranteed to be valid. To fix this
problem, add a few null checks to ignore data nodes associated
to non-implemented modules.
The side effect of this change is harmless. If a daemon receives
configuration it doesn't support (e.g. BFD peers on staticd),
that configuration will be stored but otherwise ignored. This can
only happen when using a northbound client like gRPC, as the CLI
will never send to a daemon a command it doesn't support. This
minor problem should go away in the long run as FRR migrates to
a centralized management model, at which point the YANG-modeled
configuration of all daemons will be maintained in a single place.
Finally, update some daemons to stop implementing YANG modules
they don't need to (i.e. revert 1b741a01c and a74b47f5).
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
In case of config rollback is enabled,
record northbound transaction based on a control flag.
The actual frr daemons would set the flag to true via
nb_init from frr_init.
This will allow test daemon to bypass recording
transacation to db.
Signed-off-by: Chirag Shah <chirag@nvidia.com>
During the prep phase to apply a northbound commit, if no changes were
detected make sure we fill the error message buffer to explain this.
Signed-off-by: Quentin Young <qlyoung@nvidia.com>
While a configuration transaction can't be rejected once it reaches
the APPLY phase, we should allow NB callbacks to generate error
or warning messages when a configuration change is being applied.
That should be useful, for example, to return warnings back to
the user informing that the applied configuration has some kind of
inconsistency or is missing something in order to be effectively
activated. The infrastructure for this was already present, but the
northbound layer was ignoring all errors/warnings generated during
the apply/abort phases instead of returning them to the user. This
commit changes that.
In the gRPC plugin, extend the Commit() RPC adding a new
"error_message" field to the response type. This is necessary to
allow errors/warnings to be returned even when the commit operation
succeeds (since grpc::Status::OK doesn't support error messages
like the other status codes).
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
All userdata pointers need to be rekeyed to their new xpaths, not just
the one associated with the dnode being moved.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
Each northbound callback has a set of valid return values, some of
which might depend on the transaction phase. The valid return values
for each callback are documented in the northbound main header.
Add some code to detect when a callback returns an unexpected value
and log the occurrence. This should help us to identify and fix
such problems.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Instead of returning only error codes (e.g. NB_ERR_VALIDATION)
to the northbound clients, do better than that and also return
a human-readable error message. This should make FRR more
automation-friendly since operators won't need to dig into system
logs to find out what went wrong in the case of an error.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
The new northbound context structure contains information about
the client performing a configuration transaction. This information
will be made available to all configuration callbacks through the
args->context parameter.
The usefulness of this structure comes from the fact that it can be
used as a communication channel (both input and output) between the
northbound callbacks and the northbound clients. This can be done
through its "client_data" field which contains client-specific data.
This should cover some very specific scenarios where a northbound
callback should perform an action only if the configuration change
is coming from a given client. An example would be sending a PCEP
response to a PCE when an SR-TE policy is created or modified
through the PCEP northbound client (for that to happen, the
northbound callbacks need to have access to the PCEP request ID,
which needs to be available).
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Having a fixed set of parameters for each northbound callback isn't a
good idea since it makes it difficult to add new parameters whenever
that becomes necessary, as several hundreds or thousands of existing
callbacks need to be updated accordingly.
To remediate this issue, this commit changes the signature of all
northbound callbacks to have a single parameter: a pointer to a
'nb_cb_x_args' structure (where x is different for each type
of callback). These structures encapsulate all real parameters
(both input and output) the callbacks need to have access to. And
adding a new parameter to a given callback is as simple as adding
a new field to the corresponding 'nb_cb_x_args' structure, without
needing to update any instance of that callback in any daemon.
This commit includes a .cocci semantic patch that can be used to
update old code to the new format automatically.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
The intention here is to keep the code more organized. These wrappers
should be used by the northbound clients only, and never directly
by any YANG backend code.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Old gcc versions (< 5.x) have a bug that prevents C99 flexible
arrays from working properly on shared libraries.
We already have a hack in place to work around this problem, but it
needs to be replicated in every declaration of a frr_yang_module_info
variable within libfrr. This clearly isn't a good solution if we
consider that many more libfrr YANG modules are about to come in
the future.
This commit introduces a different workaround that operates within
the northbound layer itself, such that implementers of libfrr YANG
modules won't need to worry about this problem anymore.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Remove the xpath field from the nb_config_cb structure in order
to reduce its size. This allows the northbound to spend less time
allocating memory during the processing of large configuration
transactions.
To make this work, use yang_dnode_get_path() to obtain the xpath
from the dnode field whenever necessary.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Commit 6b5d6e2dbc changed how we order configuration callbacks
and introduced a regression in the processing of the 'apply_finish'
callbacks. Fix this.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
nb_candidate_edit() was calling both the lyd_schema_sort() and
lyd_validate() functions whenever a new node was added to the
candidate configuration. This was done to ensure the candidate
is always ready to be displayed correctly (libyang only creates
default child nodes during the validation process, and data nodes
aren't guaranteed to be ordered by default).
The problem is that the two aforementioned functions are too
expensive to be called in the northbound hot path. Instead, it makes
more sense to call them only before displaying the configuration
(in which case a recursive sort needs to be done). Introduce the
nb_cli_show_config_prepare() to achieve that purpose.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
During initialization, the northbound detects if any required
callback is missing (fatal error) or if any unneeded callback is
present (warning).
There are three callbacks, however, that should require special
handling: get_next(), get_keys() and lookup_entry().
These callbacks are normally unneeded for configuration lists. But,
if a configuration list is augmented with new state nodes by another
module, then the three callbacks mentioned above become required. In
this case, never log a warning when these callbacks are implemented
when they are not needed, since this depends on context (e.g. some
daemons might augment "frr-interface" while others don't).
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
When a configuration transaction is being performed, the northbound
uses a red-black tree to store the configuration changes that need to
be processed. The problem is that we were sorting the configuration
changes based on their XPaths (and callback priorities). This means
the original order of the changes wasn't being respected, which is
a problem for lists that use the "ordered-by user" statement. To
fix this, add a new "seq" member to the "nb_config_cb" structure
so that we can preserve the order of the configuration changes as
told by libyang.
Since none of the FRR modules use "ordered-by user" lists so far,
no daemon was affected by this problem.
Reported-by: Martin Winter <mwinter@opensourcerouting.org>
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
When updating the XPath during the iteration of operational data,
include the namespace of the augmenting module when necessary.
Reported-by: Quentin Young <qlyoung@cumulusnetworks.com>
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Adding a lock to protect the global running configuration doesn't
help much since the FRR daemons are not prepared to process
configuration changes in a pthread that is not the main one (a
whole lot of new protections would be necessary to prevent race
conditions).
This means the lock added by commit 83981138 only adds more
complexity for no benefit. Remove it now to simplify the code.
All northbound clients, including the gRPC one, should either run
in the main pthread or use synchronization primitives to process
configuration transactions in the main pthread.
This reverts commit 83981138fe.
This callback can be used to validate subsections of the
configuration being committed before validating the configuration
changes themselves. It's useful to perform more complex validations
that depend on the relationship between multiple nodes.
Only YANG-level validation (performed by libyang) and the
NB_EV_VALIDATE validation (that can be used to validate individual
configuration changes) proved to be insufficient in some cases.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
frr_with_mutex(...) { ... } locks and automatically unlocks the listed
mutex(es) when the block is exited. This adds a bit of safety against
forgetting the unlock in error paths & co. and makes the code a slight
bit more readable.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
When using the LYD_PATH_OPT_NOPARENTRET flag, lyd_new_path() returns
the path-referenced node instead of the first created node. This
flag wasn't available in libyang 0.16-r1 so we couldn't use it
before. Use it now to simplify the code where possible.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
libyang-0.16-r3 contains a commit[1] that changed the autodelete
behavior of subtrees when validating data. A few FRR commands were
affected by this change since they relied on the old autodelete
behavior.
To fix these commands, use the LYD_OPT_WHENAUTODEL flag when
validating data to restore the old autodelete behavior (which adds
a lot of convenience for us).
[1] https://github.com/CESNET/libyang/commit/bbc43b1b4
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
gcc is complaing about this with --enable-dev and --enable-werror:
In function 'nb_log_callback',
inlined from 'nb_transaction_apply_finish' at lib/northbound.c:1106:4:
lib/northbound.c:777:2: error: '%s' directive argument is null [-Werror=format-overflow=]
777 | zlog_debug(
| ^~~~~~~~~~~
778 | "northbound callback: event [%s] op [%s] xpath [%s] value [%s]",
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
779 | nb_event_name(event), nb_operation_name(operation), xpath,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
780 | value);
| ~~~~~~
CC lib/ringbuf.lo
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
It doesn't make much sense for a hash function to modify its argument,
so const the hash input.
BGP does it in a couple places, those cast away the const. Not great but
not any worse than it was.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
The upcoming gRPC-based northbound plugin will run on a separate
pthread, and it will need to have access to the running configuration
global variable. Introduce a rw-lock to control concurrent access
to the running configuration. Add the lock inside the "nb_config"
structure so that it can be used to protect candidate configurations
as well (this might be necessary depending on the threading scheme
of future northbound plugins).
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
The ability to lock the running configuration to prevent other users
from changing it is a very important one. We already supported
the "configure exclusive" command but the lock was applied to
the CLI users only (other clients like ConfD could still commit
configuration transactions, ignoring the CLI lock). This commit
introduces a global lock for the running configuration that is
shared by all northbound clients, and provides a public API to
manipulate it. This way other northbound clients will also be able
to lock/unlock the running configuration if required (the upcoming
gRPC northbound plugin will have RPCs for that).
NOTE: this is a management-level lock for the running configuration,
not to be confused with low-level locks used to avoid data races.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Certain operations, like removing non-presence containers or
modifying list keys, are not considered to be valid from the
perspective of the northbound layer. This is because we want to
implement a minimum set of northbound configuration callbacks and
use them to process all possible configuration changes.
The removal of a np-container [1], for example, can be processed by
calling the "delete" callback of all of its child nodes (recursion
is used for np-container child nodes). Similarly, the modification
of a list key can be processed as if the corresponding list entry
was removed and readded with updated key values. This strategy saves
us the burden of implementing lots of extra configuration callbacks.
That said, the nb_operation_is_valid() function shouldn't be used
for anything other than checking which callbacks are valid for
which YANG nodes. Using it in the nb_candidate_edit() function
is inappropriate as we want as much flexibility as possible when
editing a candidate configuration. We should allow CLI commands,
for example, to remove np-containers (the northbound layer will then
figure out which callbacks need to be called when this candidate
is committed). Remove the check.
[1] We can't do the same for presence containers since they have a
"create" callback associated with them.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
flog() is a small wrapper around zlog() that can be useful in a
few places to reduce code duplication.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Move call to nb_db_init() from nb_init() to frr_init() so that only
the FRR daemons will initialize the northbound database. This should
fix a few warnings when running some unit tests.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Introduce a hash table to keep track of user pointers associated
to configuration entries. The previous strategy was to embed
the user pointers inside libyang data nodes, but this solution
incurred a substantial performance overhead. The user pointers
embedded in candidate configurations could be lost while the
configuration was being edited, so they needed to be regenerated
before the candidate could be committed. This was done by the
nb_candidate_restore_priv_pointers() function, which was extremely
expensive for large configurations. The new hash table solves this
performance problem.
The yang_dnode_[gs]et_entry() functions were renamed and moved from
yang.[ch] to northbound.[ch], which is a more appropriate place
for them. This patch also introduces the nb_running_unset_entry()
function, the counterpart of nb_running_set_entry() (unsetting
user pointers was done automatically before, now it needs to be
done manually).
As a consequence of these changes, we shouldn't need support for
libyang private pointers anymore (-DENABLE_LYD_PRIV=ON). But it's
probably a good idea to keep requiring this feature as we might
need it in the future for other things (e.g. disable configuration
settings without removing them).
Fixes#4136.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Split the "debug northbound" command into the following commands:
* debug northbound callbacks configuration
* debug northbound callbacks state
* debug northbound callbacks rpc
* debug northbound notifications
* debug northbound events
* debug northbound client confd
* debug northbound client sysrepo
If "debug northbound" is entered alone, all of its suboptions
are enabled.
This commit also adds code to debug state/rpc callbacks and
notifications (only configuration callbacks were logged before).
Use the debugging infrastructure from "lib/debug.h" in order to
benefit from its facilities (e.g. MT-safe debugging) and avoid
code duplication.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Non-presence containers don't have "destroy" callbacks. So, once
a np-container is deleted, we need to call the "destroy" callbacks
of its child nodes instead.
This commit doesn't fix any real problem as of now since all
np-containers from the FRR YANG modules contain or one more mandatory
child nodes, so they can't be deleted (libyang will add missing
np-containers when validating data). Nevertheless, upcoming YANG
modules should benefit from this change.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
This is just a small refactoring to reduce code duplication. No
behavior changes intended.
Signed-off-by: Renato Westphal <renato@opensourcerouting.org>
Change the northbound lib operation from DELETE to DESTROY;
make the required changes in the users of the northbound, in
the cli, rip, ripng, and isis.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
