Thread statistics are collected and stored in a hashtable shared across
threads, but while the hashtable itself is protected by a mutex, the
records themselves were not being updated safely. Change all thread
history collection to use atomic operations.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
The following types are nonstandard:
- u_char
- u_short
- u_int
- u_long
- u_int8_t
- u_int16_t
- u_int32_t
Replace them with the C99 standard types:
- uint8_t
- unsigned short
- unsigned int
- unsigned long
- uint8_t
- uint16_t
- uint32_t
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
There are some observed instances where we end up trying to cancel a rw
job based on a file descriptor that we don't have a reference on. The
specific cancel function for rw jobs assumes it's called with a file
descriptor that is valid within pollfds and will cause a segmentation
fault by buffer overrun if this is not the case.
Instead log it and move on. Since the fd does not exist this should
patch over the buggy behavior and provide additional information to help
in finding the root cause.
Reviewed-by: Donald Sharp <sharpd@cumulusnetworks.com>
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
When we remove a thread from a pqueue, use the saved
index to go to the correct spot immediately instead of
having to search the whole queue for it.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
When displaying thread cpu data, display unsigned instead
of signed data when we get really really really large
numbers of invocations.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
list_free is occassionally being used to delete the
list and accidently not deleting all the nodes.
We keep running across this usage pattern. Let's
remove the temptation and only allow list_delete
to handle list deletion.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Convert the list_delete(struct list *) function to use
struct list **. This is to allow the list pointer to be nulled.
I keep running into uses of this list_delete function where we
forget to set the returned pointer to NULL and attempt to use
it and then experience a crash, usually after the developer
has long since left the building.
Let's make the api explicit in it setting the list pointer
to null.
Cynical Prediction: This code will expose a attempt
to use the NULL'ed list pointer in some obscure bit
of code.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
list_delete does not set the list pointer to NULL
Thus when we accidently use it later we happily write
off into lala land instead of crashing imediately
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
1) Some hash key functions where converting pointers
directly to a 32 bit value via downcasting. Pointers
are 64 bit on a majority of our platforms.
2) Some hashes were being created with 256 entries,
downsize the hash creation size to more appropriate
values.
3) Add hash names to hash creation so we can watch
the hash via 'show debugging hashtable'
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
* Fix bad format specifier in thread.[ch]
* Move PRINTF_ATTRIBUTE macro to zebra.h
* Use PRINTF_ATTRIBUTE on termtable printers
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
Add support for naming pthreads. Also, note that we don't have any
records yet if that's the case.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
This patch fixes up show thread commands so that they know about
and operate on all extant thread_masters, since we can now have multiple
running in any given application.
This change also eliminates a heap use after free that appears when
using a single cpu_record shared among multiple threads. Since struct
thread's have pointers to bits of memory that are freed when the global
statistics hash table is freed, later accesses are invalid. By moving
the stats hash to be unique to each thread_master this problem is
sidestepped.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
* Account for the pipe poker in poll() by explicitly returning NULL when
we have no events, timers or file descriptors to work with
* Add a comment explaining exactly what we are doing and why
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
* Update pollfds copy as well as the original
* Keep array count for copy in thread_master
* Remove last remnants of POLLHUP in .events field
* Remove unused snmpcount (lolwut)
* Improve docs
* Add missing do_thread_cancel() call in thread_cancel_event()
* Change thread_fetch() to always enter poll() to avoid starving i/o
* Remember to free up cancel_req when destroying thread_master
* Fix dereference of null pointer
* Fix dead store to timeval
* Fix missing condition for condition variable :-)
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
This patch implements an MT-safe version of thread_cancel() in
thread_cancel_async(). Behavior as follows:
* Cancellation requests are queued into a list
* Cancellation requests made from the same pthread as the thread_master
owner are serviced immediately (thread_cancel())
* Cancellation requests made from a separate pthread are queued and the
call blocks on a condition variable until the owning pthread services
the request, at which point the condition variable is signaled and
execution continues (thread_cancel_async())
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
it's just an alias for a millisecond timer used in exactly nine places
and serves only to complicate
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
If fd_poll() is called with no file descriptors, an incorrect check in
the function prelude causes it to return instantly; for a thread that
wishes to poll but has no file descriptors, this results in busy
waiting. Desired behavior is to block.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
a bunch of pollfds can cause a stack overflow when using a stack
allocated buffer...silly me...
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
When scheduling a task onto a thread master owned by another pthread, we
need to lock the thread master's mutex. However, if the pthread which
owns that thread master is in poll(), we could be stuck waiting for a
very long time. To solve this, we copy all data poll() needs and unlock
during poll(). To break the target pthread out of poll(), thread_master
has gained a pipe whose reading end is passed into poll(). After an event
that requires immediate action by the target pthread, a byte is written
into the pipe in order to wake it up.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
[DL: split off from select() removal]
poll() is present on every supported platform and does not have an upper
limit on file descriptors.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
[DL: split off from AWAKEN() change]
Allow some more flexibility in case callers wish to manage their own
thread pointers and don't require or don't want the thread to keep a
back reference to its holding pointer.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
The FSF's address changed, and we had a mixture of comment styles for
the GPL file header. (The style with * at the beginning won out with
580 to 141 in existing files.)
Note: I've intentionally left intact other "variations" of the copyright
header, e.g. whether it says "Zebra", "Quagga", "FRR", or nothing.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
Passing stack value to thread_add_* causes thread->ref to become an
invalid pointer when the value goes out of scope
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
When scheduling a thread, the scheduling function returns a pointer to
the struct thread that was placed on one of the scheduling queues in the
associated thread master. This pointer is used to check whether or not
the thread is scheduled, and is passed to thread_cancel() should the
daemon need to cancel that particular task.
The thread_fetch() function is called to retrieve the next thread to
execute. However, when it returns, the aforementioned pointer is not
updated. As a result, in order for the above use cases to work, every
thread handler function must set the associated pointer to NULL. This is
bug prone, and moreover, not thread safe.
This patch changes the thread scheduling functions to return void. If
the caller needs a reference to the scheduled thread, it must pass in a
pointer to store the pointer to the thread struct in. Subsequent calls
to thread_cancel(), thread_cancel_event() or thread_fetch() will result
in that pointer being nulled before return. These operations occur
within the thread_master critical sections.
Overall this should avoid bugs introduced by thread handler funcs
forgetting to null the associated pointer, double-scheduling caused by
overwriting pointers to currently scheduled threads without performing a
nullity check, and the introduction of true kernel threads causing race
conditions within the userspace threading world.
Also removes the return value for thread_execute since it always returns
null...
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
The way thread.c is written, a caller who wishes to be able to cancel a
thread or avoid scheduling it twice must keep a reference to the thread.
Typically this is done with a long lived pointer whose value is checked
for null in order to know if the thread is currently scheduled. The
check-and-schedule idiom is so common that several wrapper macros in
thread.h existed solely to provide it.
This patch removes those macros and adds a new parameter to all
thread_add_* functions which is a pointer to the struct thread * to
store the result of a scheduling call. If the value passed is non-null,
the thread will only be scheduled if the value is null. This helps with
consistency.
A Coccinelle spatch has been used to transform code of the form:
if (t == NULL)
t = thread_add_* (...)
to the form
thread_add_* (..., &t)
The THREAD_ON macros have also been transformed to the underlying
thread.c calls.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
Rename HAVE_POLL to HAVE_POLL_CALL, when compiling with
snmp and poll enabled this was causing issues.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
This change adds three fields to thread_master and associated code to
use them. The fields are:
* long selectpoll_timeout
This is a millisecond value that, if nonzero, will override the
internally calculated timeout for select()/poll(). -1 indicates
nonblocking while a positive value indicates the desired timeout in
milliseconds.
* bool spin
This indicates whether a call to thread_fetch() should result in a loop
until work is available. By default this is set to true, in order to
keep the default behavior. In this case a return value of NULL indicates
that a fatal signal was received in select() or poll(). If it is set to
false, thread_fetch() will return immediately. NULL is then an
acceptable return value if there is no work to be done.
* bool handle_signals
This indicates whether or not the pthread that owns the thread master
is responsible for handling signals (since this is an MT-unsafe
operation, it is best to have just the root thread do it). It is set to
true by default. Non-root pthreads should set this to false.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
Fixes a few insufficient critical sections. Adds back locking for
thread_cancel(), since while thread_cancel() is only safe to call from
the pthread which owns the thread master due to races involving
thread_fetch() modifying thread master's ready queue, we still need
mutual exclusion here for all of the other public thread.c functions to
maintain their MT-safety.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
This change introduces synchronization mechanisms to thread.c in order
to allow safe concurrent use.
Thread.c should now be threadstafe with respect to:
* struct thread
* struct thread_master
Calls into thread.c for operations upon data of this type should not
require external synchronization.
Signed-off-by: Quentin Young <qlyoung@cumulusnetworks.com>
In the case where we are using select as
the operator *and* we call
funcname_thread_add_read_write *and* the
fd is already set, we would overwrite
the read/write direction to always be READ.
Clearly this was a bad idea.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Since time is no longer cached, if we schedule something with zero
timeout, it will automatically be negative by the time we reach the
event loop.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
monotime_since() does exactly the same thing.
... and timeval_elapsed is now private to lib/thread.c
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
Fix the display of 'show thread cpu' to keep track
of the number of active threads and to display that
information.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
lib: Fix thread_execute_crash
This commit does these things:
1) Make thread_add_unuse own the setting of THREAD_UNUSED.
2) Move thread->hist finding to to thread_get.
We are storing the thread->hist even when the thread
is on the unused. This means that we check to see
if the funcname or func have changed and we get new
history. Else we've probably just retrieved the last
unused which has the same func/funcanme. This is
a common practice to do THREAD_OFF/THREAD_ON in
quick succession.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com.
This moves all install_element calls into the file where the DEFUNs are
located. This fixes several small related bugs:
- ospf6d wasn't installing a "no interface FOO" command
- zebra had a useless copy of "interface FOO"
- pimd's copy of "interface FOO" was not setting qobj_index, which means
"description LINE" commands would fail with an error
The next commit will do the actual act of making "foo_cmd" static.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
Since we have autoconf results from a wide swath of target platforms, we
can go remove checks that have the same result on all systems.
This also removes several "fallback" implementations of functions that,
at some point in the history, weren't available on all target platforms.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
When displaying thread time for long running/busy
protocols, the space allocated may not be sufficient.
Allow the runtime to take a bit more space.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
This is a rather large mechanical commit that splits up the memory types
defined in lib/memtypes.c and distributes them into *_memory.[ch] files
in the individual daemons.
The zebra change is slightly annoying because there is no nice place to
put the #include "zebra_memory.h" statement.
bgpd, ospf6d, isisd and some tests were reusing MTYPEs defined in the
library for its own use. This is bad practice and would break when the
memtype are made static.
Acked-by: Vincent JARDIN <vincent.jardin@6wind.com>
Acked-by: Donald Sharp <sharpd@cumulusnetworks.com>
[CF: rebased for cmaster-next]
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
Signed-off-by: Christian Franke <chris@opensourcerouting.org>
The regular expression for finding DEFUN/ALIAS in
extract.pl looks for "DEFUN (" or "ALIAS (" if
the *.c file does not have this then it will just
silently ignore the cli.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
AgentX fd/timeout handling is rather hackishly monkeyed into thread.c.
Replace with code that uses plain thread_* functions.
NB: Net-SNMP's API rivals Quagga's in terms of age and absence of
documentation. netsnmp_check_outstanding_agent_requests() in particular
seems to be unused and is therefore untested.
The most useful documentation on this is actually the blog post Vincent
Bernat wrote when he originally integrated this into lldpd and Quagga:
https://vincent.bernat.im/en/blog/2012-snmp-event-loop.html
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
Another zoo extension, this adds a timer scheduling function that takes
a struct timeval argument (which is actually what the wrappers boil down
to, yet it's not exposed...)
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
QUAGGA_CLK_REALTIME and QUAGGA_CLK_REALTIME_STABILISED aren't used
anywhere in the code. Remove. The enum is kept to avoid having to
change the calls everywhere.
Same applies to the workaround code for systems that don't have a
monotonic clock. None of the systems Quagga works on fall into that
category; Linux, BSD and Solaris all do clock_gettime, for OSX we have
mach_absolute_time() - that covers everything.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
ospf->lsa_refresher_started is only used in relative timing to itself;
replace with monotonic clock which is appropriate for this.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
- HAVE_POLL is overloaded by net-snmp
- missing includes
- ospf6_snmp converted to vrf_iflist()
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
Two Fixes:
1) When a fd has both read and write as a .events.
(POLLHUP | POLLIN | POLLOUT) and a
thread_cancel_read_write call is executed
from a protocol, the code was blindly removing
the fd from consideration at all.
2) POLLNVAL was being evaluated before POLLIN|POLLOUT
were being evaluated. While I didn't see a case
of POLLNVAL being included with other .revent flags
I decided to move the POLLNVAL and POLLHUP handling
to the same section of code.
Additionally the function thread_cancel_read_write
was poorly named and let me to poorly implement
the poll version of it. I've renamed the function
thread_cancel_read_or_write in an attempt to
make this problem moot in the future.
Ticket: CM-11027
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
(cherry picked from commit f6da66a913)
now that we know what thread we're currently executing, let's add that
information to SEGV / assert backtraces.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
(cherry picked from commit 615f9f18fc025757a255f936748fc1e86e922783)
the library's thread scheduling functions keep track of the thread
function's name, so far so good. However, copying the compiler-provided
constant into a buffer inside the thread structure is plain useless.
Also, strip_funcname() was trying to support something that never
happens.
Instead, let's use some bytes here to track where threads are scheduled
from. Another commit will print that information on crashes.
Ripping out useless stuff: -64 bytes in the thread structure
Re-add as const ptr: +8 bytes
Extra debug info: +12 bytes
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
(cherry picked from commit 3493b7731b750cbc62f00be94b624a08ccccf0b2)
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Reviewed-by: Don Slice <dslice@cumulusnetworks.com>
Reviewed-by: Daniel Walton <dwalton@cumulusnetworks.com>
poll returns the number of revents that we need to handle
in the array. revent is a bit field of events that need
to be handled. thread.c was treating each sub item in the
bitfield as a separate item to handle.
As such the loop over the pollfds would quit early
sometimes.
Ticket: CM-10077
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Reviewed-by: Don Slice <dslice@cumulusnetworks.com>
The pollfds was being resized if the # of fds grew to
be more than the original array size. Just size it
once.
Ticket: CM-10077
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Reviewed-by: Don Slice <dslice@cumulusnetworks.com>
This patch originated w/ Hannes Hofer <hhofer@barracuda.com>.
I've taken the patch fixed some bugs and reworked the code
to allow both poll and select to be choosen at compile time.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
thread_process_fd is looping over the read and write
fd's separately. There is no need to do this individually.
loop over both the read and write fdset's at the same time.
This will improve select processing performance, especially
for large data sets.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Both the read and write functions used the same code
slightly modified for reading and writing. Combine this
code together.
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Abstract FD set operations so that we can eventually
choose what type of select/poll operation that
we want to use.
Signed-off-by: Donald Sharp <sharpd@cumulusnetowkrs.com>
Ticket: CM-4109
Reviewed-by: CCR-3414
Testing: See bug
Fixup of these memory issues:
(A) peer->clear_node_queue was accidently removed. Add back in.
(B) Clean up bm->process_main_queue and bm->process_rsclient_queue initialization
(C) Some memory leaks
(D) Clean up unused threads
Ticket : CM-6300
Reviewed by : CCR-3049
Testing : Manual
With arrays, a thread corresponding to given fd is looked up in constant time
versus the linear time taken for list traversals.
Support stubby and totally stubby areas in OSPFv3
Signed-off-by: Dinesh G Dutt <ddutt at cumulusnetworks.com>
Reviewed-by: Pradosh Mohapatra <pmohapat at cumulusnetworks.com>
- Allow work queues to specify the yield duration for corresponding background thread
- Support using specified yield duration in thread yielding
- During work queue processing, if using a single list element with a meta-queue
(like done in Zebra), do not exit after each element is processed, instead
update the next-node upon a WQ_REQUEUE so that the WQ processing continues
and is terminated by the yield logic.
- Enhance work queue debug output
Compute and display SPF execution statistics
Detailed SPF statistics, all around time spent executing various pieces of SPF
such as the SPF algorithm itself, installing routes, pruning unreachable networks
etc.
Reason codes for firing up SPF are:
R - Router LSA, N - Network LSA, S - Summary LSA, ABR - ABR status change,
ASBR - ASBR Status Change, AS - ASBR Summary, M - MaxAge
Signed-off-by: Dinesh G Dutt <ddutt@cumulusnetworks.com>
Reviewed-by: JR Rivers <jrrivers@cumulusnetworks.com>
Reviewed-by: Scott Feldman <sfeldma@cumulusnetworks.com>
Reviewed-by: Ayan Banerjee <ayan@cumulusnetworks.com>
Quagga sources have inherited a slew of Page Feed (^L, \xC) characters
from ancient history. Among other things, these break patchwork's
XML-RPC API because \xC is not a valid character in XML documents.
Nuke them from high orbit.
Patches can be adapted simply by:
sed -e 's%^L%%' -i filename.patch
(you can type page feeds in some environments with Ctrl-V Ctrl-L)
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
By forcing the hash table size to be a power of 2, a potentially
expensive divide can be replaced by a mask operation. Almost all
usage of the hash table was using default size of 1024. Only places
with different size was thread library (1011) and bgp aspath.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
There is no Posix CLOCK_MONOTONIC in Darwin, but monotonically
increasing clock can be implemented using mach_absolute_time().
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
--enable-snmp will enable AgentX support in Quagga. SMUX is still here
and can be enabled with --enable-snmp=smux. AgentX support can be
enabled with "agentx" in configuration file. As for SMUX, this command
is not understood by vtysh. It can be disabled with "no agentx",
though there is no real use of this since this command cannot be used
with vtysh.
If "agentx" and "no agentx" command were added to vtysh, it would not
be possible to disable agentx support after enabling it because
NetSNMP does not expose the appropriate methods for this.
The internals of AgentX are hidden by NetSNMP. Therefore, we don't
have a file descriptor to add to the threading system. We do not have
the timers to set either. Therefore, the event loop is modified to
make use of snmp_select_info() from NetSNMP.
Traps are not supported yet.
This avoids memory heap fragmentation and imposses less load on the
system memory allocator.
* thread.h: FUNCNAME_LEN defined to 64 (ISO C99 says max 63)
Signed-off-by: Jorge Boncompte [DTI2] <jorge@dti2.net>
[changed FUNCNAME_LEN to a less arbitrary value]
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
thread_trim_head() already checks that the list is not empty.
Signed-off-by: Jorge Boncompte [DTI2] <jorge@dti2.net>
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
* thread.c: It's only temporarily used in thread_call() to calculate the
diffs. Saves 80 bytes per copy.
Signed-off-by: Jorge Boncompte [DTI2] <jorge@dti2.net>
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
ospfd was crashing some times on neighbour going down. The cause was that
ospf_nsm_event() was accessing already freed memory in ospf_nbr_delete()
call from ospf_nsm_event().
What happens is that since commit b5043aab (lib: fix incorrect thread
list...) now a thread can be on the event and ready lists but
thread_cancel_event() doesn't account for that.
* thread.c: (thread_cancel_event) loop on the ready list too to cancel
pending events.
Signed-off-by: Jorge Boncompte [DTI2] <jorge@dti2.net>
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
* thread.c: (thread_timer_process,thread_process) thread_list_delete nulls
thread->next. Loops need to save next first, or will only process the head.
Problem noted by Lou Berger <lberger@labn.net>.
* (general) Move functions in headers into files, to be compiled into
shared object files. Remove inline qualifier from functions. Let the
compiler do the work.
* (general) this can be useful when investigating thread latency problems,
when you don't want to have to restart a daemon between tests.
* thread.c: (cpu_record_(hash_)clear) wipe the stored thread cpu history
data, according to the filter, similar to the vty print code.
(clear_thread_cpu_cmd) new command to clear data.
* thread.h: export new command
* command.c: install it
* thread.c: (thread_fetch) the current scheduler will service events
indefinitely, ignoring I/O and timers, so long as there are events.
In other words, events can crowd out I/O and timers. In theory this
shouldn't be a huge problem as events are generated only by timers
and I/O, however in practice it means normal-load behaviour is not as
useful a predictor of high-load behaviour as it should be.
Fix this by considering all the kinds of threads, in every run of the
scheduler. For any given run, we prioritise events, however across
runs the scheduler should be fair.
This has been observed to give more stable inter-packet times in
testing of ospfd (i.e. lower std-dev).
(thread_process) new heler to queue all the given threads onto the ready
list
* thread.h: (struct cpu_thread_history.funcname) malloc gets called on this
so it should not be const
* thread.c: (cpu_record_print) cast the (const char *) to (char *), this
function knows what it's doing (free wont be called on it).
I've spent the last several weeks working on stability fixes to bgpd.
These patches fix all of the numerous crashes, assertion failures, memory
leaks and memory stomping I could find. Valgrind was used extensively.
Added new function bgp_exit() to help catch problems. If "debug bgp" is
configured and bgpd exits with status of 0, statistics on remaining
lib/memory.c allocations are printed to stderr. It is my hope that other
developers will use this to stay on top of memory issues.
Example questionable exit:
bgpd: memstats: Current memory utilization in module LIB:
bgpd: memstats: Link List : 6
bgpd: memstats: Link Node : 5
bgpd: memstats: Hash : 8
bgpd: memstats: Hash Bucket : 2
bgpd: memstats: Hash Index : 8
bgpd: memstats: Work queue : 3
bgpd: memstats: Work queue item : 2
bgpd: memstats: Work queue name string : 3
bgpd: memstats: Current memory utilization in module BGP:
bgpd: memstats: BGP instance : 1
bgpd: memstats: BGP peer : 1
bgpd: memstats: BGP peer hostname : 1
bgpd: memstats: BGP attribute : 1
bgpd: memstats: BGP extra attributes : 1
bgpd: memstats: BGP aspath : 1
bgpd: memstats: BGP aspath str : 1
bgpd: memstats: BGP table : 24
bgpd: memstats: BGP node : 1
bgpd: memstats: BGP route : 1
bgpd: memstats: BGP synchronise : 8
bgpd: memstats: BGP Process queue : 1
bgpd: memstats: BGP node clear queue : 1
bgpd: memstats: NOTE: If configuration exists, utilization may be expected.
Example clean exit:
bgpd: memstats: No remaining tracked memory utilization.
This patch fixes bug #397: "Invalid free in bgp_announce_check()".
This patch fixes bug #492: "SIGBUS in bgpd/bgp_route.c:
bgp_clear_route_node()".
My apologies for not separating out these changes into individual patches.
The complexity of doing so boggled what is left of my brain. I hope this
is all still useful to the community.
This code has been production tested, in non-route-server-client mode, on
a linux 32-bit box and a 64-bit box.
Release/reset functions, used by bgp_exit(), added to:
bgpd/bgp_attr.c,h
bgpd/bgp_community.c,h
bgpd/bgp_dump.c,h
bgpd/bgp_ecommunity.c,h
bgpd/bgp_filter.c,h
bgpd/bgp_nexthop.c,h
bgpd/bgp_route.c,h
lib/routemap.c,h
File by file analysis:
* bgpd/bgp_aspath.c: Prevent re-use of ashash after it is released.
* bgpd/bgp_attr.c: #if removed uncalled cluster_dup().
* bgpd/bgp_clist.c,h: Allow community_list_terminate() to be called from
bgp_exit().
* bgpd/bgp_filter.c: Fix aslist->name use without allocation check, and
also fix memory leak.
* bgpd/bgp_main.c: Created bgp_exit() exit routine. This function frees
allocations made as part of bgpd initialization and, to some extent,
configuration. If "debug bgp" is configured, memory stats are printed
as described above.
* bgpd/bgp_nexthop.c: zclient_new() already allocates stream for
ibuf/obuf, so bgp_scan_init() shouldn't do it too. Also, made it so
zlookup is global so bgp_exit() can use it.
* bgpd/bgp_packet.c: bgp_capability_msg_parse() call to bgp_clear_route()
adjusted to use new BGP_CLEAR_ROUTE_NORMAL flag.
* bgpd/bgp_route.h: Correct reference counter "lock" to be signed.
bgp_clear_route() now accepts a bgp_clear_route_type of either
BGP_CLEAR_ROUTE_NORMAL or BGP_CLEAR_ROUTE_MY_RSCLIENT.
* bgpd/bgp_route.c:
- bgp_process_rsclient(): attr was being zero'ed and then
bgp_attr_extra_free() was being called with it, even though it was
never filled with valid data.
- bgp_process_rsclient(): Make sure rsclient->group is not NULL before
use.
- bgp_processq_del(): Add call to bgp_table_unlock().
- bgp_process(): Add call to bgp_table_lock().
- bgp_update_rsclient(): memset clearing of new_attr not needed since
declarationw with "= { 0 }" does it. memset was already commented
out.
- bgp_update_rsclient(): Fix screwed up misleading indentation.
- bgp_withdraw_rsclient(): Fix screwed up misleading indentation.
- bgp_clear_route_node(): Support BGP_CLEAR_ROUTE_MY_RSCLIENT.
- bgp_clear_node_queue_del(): Add call to bgp_table_unlock() and also
free struct bgp_clear_node_queue used for work item.
- bgp_clear_node_complete(): Do peer_unlock() after BGP_EVENT_ADD() in
case peer is released by peer_unlock() call.
- bgp_clear_route_table(): Support BGP_CLEAR_ROUTE_MY_RSCLIENT. Use
struct bgp_clear_node_queue to supply data to worker. Add call to
bgp_table_lock().
- bgp_clear_route(): Add support for BGP_CLEAR_ROUTE_NORMAL or
BGP_CLEAR_ROUTE_MY_RSCLIENT.
- bgp_clear_route_all(): Use BGP_CLEAR_ROUTE_NORMAL.
Bug 397 fixes:
- bgp_default_originate()
- bgp_announce_table()
* bgpd/bgp_table.h:
- struct bgp_table: Added reference count. Changed type of owner to be
"struct peer *" rather than "void *".
- struct bgp_node: Correct reference counter "lock" to be signed.
* bgpd/bgp_table.c:
- Added bgp_table reference counting.
- bgp_table_free(): Fixed cleanup code. Call peer_unlock() on owner if
set.
- bgp_unlock_node(): Added assertion.
- bgp_node_get(): Added call to bgp_lock_node() to code path that it was
missing from.
* bgpd/bgp_vty.c:
- peer_rsclient_set_vty(): Call peer_lock() as part of peer assignment
to owner. Handle failure gracefully.
- peer_rsclient_unset_vty(): Add call to bgp_clear_route() with
BGP_CLEAR_ROUTE_MY_RSCLIENT purpose.
* bgpd/bgp_zebra.c: Made it so zclient is global so bgp_exit() can use it.
* bgpd/bgpd.c:
- peer_lock(): Allow to be called when status is "Deleted".
- peer_deactivate(): Supply BGP_CLEAR_ROUTE_NORMAL purpose to
bgp_clear_route() call.
- peer_delete(): Common variable listnode pn. Fix bug in which rsclient
was only dealt with if not part of a peer group. Call
bgp_clear_route() for rsclient, if appropriate, and do so with
BGP_CLEAR_ROUTE_MY_RSCLIENT purpose.
- peer_group_get(): Use XSTRDUP() instead of strdup() for conf->host.
- peer_group_bind(): Call bgp_clear_route() for rsclient, and do so with
BGP_CLEAR_ROUTE_MY_RSCLIENT purpose.
- bgp_create(): Use XSTRDUP() instead of strdup() for peer_self->host.
- bgp_delete(): Delete peers before groups, rather than after. And then
rather than deleting rsclients, verify that there are none at this
point.
- bgp_unlock(): Add assertion.
- bgp_free(): Call bgp_table_finish() rather than doing XFREE() itself.
* lib/command.c,h: Compiler warning fixes. Add cmd_terminate(). Fixed
massive leak in install_element() in which cmd_make_descvec() was being
called more than once for the same cmd->strvec/string/doc.
* lib/log.c: Make closezlog() check fp before calling fclose().
* lib/memory.c: Catch when alloc count goes negative by using signed
counts. Correct #endif comment. Add log_memstats_stderr().
* lib/memory.h: Add log_memstats_stderr().
* lib/thread.c: thread->funcname was being accessed in thread_call() after
it had been freed. Rearranged things so that thread_call() frees
funcname. Also made it so thread_master_free() cleans up cpu_record.
* lib/vty.c,h: Use global command_cr. Add vty_terminate().
* lib/zclient.c,h: Re-enable zclient_free().
Whenever a thread adds an timer funcname_thread_add_timer_timeval() gets called
to add the timer. Before adding the timer a quagga_gettimeofday() call
is made to do some time house keeping. However quagga_gettimeofday() only
updates recent_time, not relative_time that is used to calculate
the alarm_time. Replace with quagga_get_relative (NULL)
2008-08-14 Stephen Hemminger <stephen.hemminger@vyatta.com>
* lib/hash.h: (struct hash) Hash comparator callback really
ought to treat storage behind arguments as constant - a compare
function with side-effects would be evil.
* */*.c: Adjust comparator functions similarly, thus fixing at least
a few compiler warnings about const qualifier being dropped.
Signed-off-by: Paul Jakma <paul@quagga.net>
2006-08-25 Paul Jakma <paul.jakma@sun.com>
* thread.c: (general) Add support for monotonic clock, it may still
jump forward by huge amounts, but should be immune to going
backwards. Fixes bug #134.
(quagga_gettimeofday_relative_adjust) helper, does what name
says - adjusts gettimeofday based relative timer.
(quagga_gettimeofday) helper to keep recent_time up to date.
(quagga_get_relative) helper, update and getch the relative
timer using gettimeofday(). POSIX CLOCK_MONOTONIC is also
supported, but the code is not enabled yet nor tested.
(quagga_real_stabilised) helper, retrieve absolute time but
stabilised so as to never decrease.
(quagga_gettime) Exported interface, analogous to POSIX
clock_gettime() in interface, supporting several clocks.
(quagga_time) Exported interface, analogous to traditional
time(), will never decrease.
(recent_relative_time) Convenience function to retrieve
relative_time timeval, similar to existing recent_time absolute
timeval, for when an approximately recent value will do.
(remainder) Update to use above helpers.
(thread_getrusage) Previously was a macro, but needs to be
a function to twiddle with thread.c private stuff.
* thread.c: Point the GETRUSAGE macro at previous function.
Export quagga_gettime, quagga_time and recent_relative_time for
general use.
2006-07-25 Paul Jakma <paul.jakma@sun.com>
* thread.h: (struct thread) Add a cache pointer to the struct
cpu_thread_history, if it is known - saving hash lookup on
each thread_call.
* thread.c: (thread_call) Cache the pointer to the
cpu_thread_history, so that future thread_calls of same
thread can avoid the hash_lookup.
* thread.c: (thread_cancel_event) the number of pending events
cancelled is potentially useful information, dont throw it away,
pass it back to the caller.
* (general) extern and static'ification of functions in code and
header.
Cleanup any definitions with unspecified arguments.
Add casts for callback assignments where the callback is defined,
typically, as passing void *, but the function being assigned has
some other pointer type defined as its argument, as gcc complains
about casts from void * to X* via function arguments.
Fix some old K&R style function argument definitions.
Add noreturn gcc attribute to some functions, as appropriate.
Add unused gcc attribute to some functions (eg ones meant to help
while debugging)
Add guard defines to headers which were missing them.
* command.c: (install_node) add const qualifier, still doesnt shut
up the warning though, because of the double pointer.
(cmp_node) ditto
* keychain.c: (key_str2time) Add GET_LONG_RANGE() macro, derived
fromn vty.h ones to fix some of the (long) < 0 warnings.
* thread.c: (various) use thread_empty
(cpu_record_hash_key) should cast to uintptr_t, a stdint.h type
* vty.h: Add VTY_GET_IPV4_ADDRESS and VTY_GET_IPV4_PREFIX so they
removed from ospfd/ospf_vty.h
* zebra.h: Move definition of ZEBRA_PORT to here, to remove
dependence of lib on zebra/zserv.h
Add wall-clock timing statistics to 'show thread cpu' output.
* thread.h: Define struct rusage_t to contain wall-clock time
and cpu time. Change GETRUSAGE macro to collect both pieces
of data. Make appropriate changes to struct cpu_thread_history
to track CPU time and real time. Change proto for
thread_consumed_time to return real and cpu time elapsed.
And declare a new global variable 'struct timeval recent_time'.
* thread.c (struct timeval recent_time): New global timestamp variable.
(timeval_adjust): If timeout is negative, set to 0 (not 10
microseconds). And remove upper bound of 1,000,000 seconds, since
this does not seem to make any sense (and it breaks
funcname_thread_add_timer_timeval).
(timeval_cmp): Should return long, not int.
(vty_out_cpu_thread_history): Show CPU time and real time.
(cpu_record_hash_print): Calculate totals for CPU and real time.
(cpu_record_print): Change 'show thread cpu' title to show CPU and
real time.
(thread_timer_remain_second): Put current time in global recent_time.
(funcname_thread_add_timer_timeval): Fix assert. Replace 2-case
switch assignment with a ternary expression. Use global recent_time
variable. Fix use of timeval_adjust (previously, the value was not
actually being adjusted).
(thread_cancel): Add missing "break" statement in case
THREAD_BACKGROUND.
(thread_timer_wait): Use global recent_time value instead of calling
gettimeofday. And there's no need to check for negative timeouts,
since timeval_subtract already sets these to zero.
(thread_timer_process): Timers are sorted, so bail out once we
encounter a timer that has not yet popped. And remove some
extraneous asserts.
(thread_fetch): Do not process foreground timers before calling
select. Instead, add them to the ready list just after the select.
Also, no need to maintain a count of the number of ready threads,
since we don't care how many there are, just whether there's
one at the head of the ready list (which is easily checked).
Stick current time in global variable recent_time to reduce
the number of calls to gettimeofday. Tighten logic for
calculating the select timeout.
(thread_consumed_time): Now returns real time and puts the elapsed
cpu time in an additional argument.
(thread_should_yield): Use real (wall-clock) time to decide whether
to yield.
(thread_call): Maintain CPU and real time statistics.
* vty.c (vty_command): For slow commands, show real and cpu time.
* thread.h: Add background thread type and thread_add_background
macro and accompanying funcname_... function.
export thread_should_yield, background threads can use it.
Lower thread yield time to 10ms, 100ms is noticeable lag and
a thread would only be /starting/ to finish sometime afterward.
* thread.c: (general) Add background thread type and schedule
nearly all thread types through the ready list for fairness.
(timeval_adjust) static qualifier missing
(vty_out_cpu_thread_history) add support for printout of
background threads
(show_thread_cpu) ditto.
(thread_master_debug) add debug of background list
(thread_master_create) fixup long line
(thread_add_unuse) add asserts for required state.
(thread_master_free) free background thread list
(funcname_thread_add_timer_timeval) make generic, able to
support arbitrary timer-like thread types.
(funcname_thread_add_timer) pass thread type to .._add_timer_timeval
(funcname_thread_add_timer_msec) ditto
(funcname_thread_add_background) Add a background thread, with an
optional millisecond delay factor, using .._add_timer_timeval.
(thread_cancel) Add background thread type.
Move the thread_list_delete common to all cases to bottom of
function, after the switch statement..
(thread_cancel_event) indent
(thread_timer_wait) Static qualifier, and make it able to cope
with arbitrary timer-like thread lists, so its of use to
background threads too.
(thread_process_fd) static qualifier. Again, make it take a list
reference rather than thread_master. Fix indentation.
(thread_timer_process) Check for ready timer-like threads in the
given list and move them on to the ready list - code originally
embedded in thread_fetch.
(thread_fetch) Schedule all threads, other than events, through
the ready list, to ensure fairness. Timer readying code moved to
thread_timer_process so it can be reused for background threads.
Remove the unneeded quagga_sigevent_process, as pointed out by
John Lin <john.ch.lin@gmail.com>.
(thread_should_yield) make this available.
* configure.ac: Added AC_ARG_ENABLE(time-check). By default,
warning messages will now be printed for threads or commands that take
longer than 5 seconds, but this configure argument can be used
to disable the checks or change the threshold.
* thread.h (thread_consumed_time): Declare new function to calculate
elapsed microseconds.
* thread.c (thread_consumed_time): Must be global not static so we
can call it from lib/vty.c:vty_command.
(thread_should_yield): Surround with `#if 0' to make clear that this
function is not currently being used anywhere.
(thread_call): If CONSUMED_TIME_CHECK is defined, print a CPU HOG
warning message if the thread takes more than CONSUMED_TIME_CHECK
microseconds.
* vty.c (vty_command): If CONSUMED_TIME_CHECK is defined, print a CPU
HOG warning message if the command takes more than CONSUMED_TIME_CHECK
microseconds.
