mirror of
https://git.proxmox.com/git/pve-kernel
synced 2025-10-04 13:34:14 +00:00
backport fix for network crashes when exiting ovs-tcpdump
A user in enterprise support reported a low (<1/50) chance of network crash (with soft lockups reported) when exiting ovs-tcpdump (a tcpdump wrapper provided by Open vSwitch) which they could only resolve by rebooting the host. After reporting the issue upstream with a reproducer [1], an OVS developer submitted a kernel patch which is now included 6.13 and some stable kernels. With this patch, the reproducer does not seem to trigger the issue anymore. Hence, backport the patch. [1] https://mail.openvswitch.org/pipermail/ovs-discuss/2025-January/053423.html Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
This commit is contained in:
parent
8f6fe4f2f0
commit
1fb1563b45
@ -0,0 +1,77 @@
|
|||||||
|
From 179893798c518361c20cb11b5b0c71cecf019ad4 Mon Sep 17 00:00:00 2001
|
||||||
|
From: Ilya Maximets <i.maximets@ovn.org>
|
||||||
|
Date: Thu, 9 Jan 2025 13:21:24 +0100
|
||||||
|
Subject: [PATCH] openvswitch: fix lockup on tx to unregistering netdev with
|
||||||
|
carrier
|
||||||
|
|
||||||
|
Commit in a fixes tag attempted to fix the issue in the following
|
||||||
|
sequence of calls:
|
||||||
|
|
||||||
|
do_output
|
||||||
|
-> ovs_vport_send
|
||||||
|
-> dev_queue_xmit
|
||||||
|
-> __dev_queue_xmit
|
||||||
|
-> netdev_core_pick_tx
|
||||||
|
-> skb_tx_hash
|
||||||
|
|
||||||
|
When device is unregistering, the 'dev->real_num_tx_queues' goes to
|
||||||
|
zero and the 'while (unlikely(hash >= qcount))' loop inside the
|
||||||
|
'skb_tx_hash' becomes infinite, locking up the core forever.
|
||||||
|
|
||||||
|
But unfortunately, checking just the carrier status is not enough to
|
||||||
|
fix the issue, because some devices may still be in unregistering
|
||||||
|
state while reporting carrier status OK.
|
||||||
|
|
||||||
|
One example of such device is a net/dummy. It sets carrier ON
|
||||||
|
on start, but it doesn't implement .ndo_stop to set the carrier off.
|
||||||
|
And it makes sense, because dummy doesn't really have a carrier.
|
||||||
|
Therefore, while this device is unregistering, it's still easy to hit
|
||||||
|
the infinite loop in the skb_tx_hash() from the OVS datapath. There
|
||||||
|
might be other drivers that do the same, but dummy by itself is
|
||||||
|
important for the OVS ecosystem, because it is frequently used as a
|
||||||
|
packet sink for tcpdump while debugging OVS deployments. And when the
|
||||||
|
issue is hit, the only way to recover is to reboot.
|
||||||
|
|
||||||
|
Fix that by also checking if the device is running. The running
|
||||||
|
state is handled by the net core during unregistering, so it covers
|
||||||
|
unregistering case better, and we don't really need to send packets
|
||||||
|
to devices that are not running anyway.
|
||||||
|
|
||||||
|
While only checking the running state might be enough, the carrier
|
||||||
|
check is preserved. The running and the carrier states seem disjoined
|
||||||
|
throughout the code and different drivers. And other core functions
|
||||||
|
like __dev_direct_xmit() check both before attempting to transmit
|
||||||
|
a packet. So, it seems safer to check both flags in OVS as well.
|
||||||
|
|
||||||
|
Fixes: 066b86787fa3 ("net: openvswitch: fix race on port output")
|
||||||
|
Reported-by: Friedrich Weber <f.weber@proxmox.com>
|
||||||
|
Closes: https://mail.openvswitch.org/pipermail/ovs-discuss/2025-January/053423.html
|
||||||
|
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
|
||||||
|
Tested-by: Friedrich Weber <f.weber@proxmox.com>
|
||||||
|
Reviewed-by: Aaron Conole <aconole@redhat.com>
|
||||||
|
Link: https://patch.msgid.link/20250109122225.4034688-1-i.maximets@ovn.org
|
||||||
|
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||||||
|
(cherry picked from commit 47e55e4b410f7d552e43011baa5be1aab4093990)
|
||||||
|
Signed-off-by: Friedrich Weber <f.weber@proxmox.com>
|
||||||
|
---
|
||||||
|
net/openvswitch/actions.c | 4 +++-
|
||||||
|
1 file changed, 3 insertions(+), 1 deletion(-)
|
||||||
|
|
||||||
|
diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c
|
||||||
|
index 101f9a23792c..896f50387347 100644
|
||||||
|
--- a/net/openvswitch/actions.c
|
||||||
|
+++ b/net/openvswitch/actions.c
|
||||||
|
@@ -930,7 +930,9 @@ static void do_output(struct datapath *dp, struct sk_buff *skb, int out_port,
|
||||||
|
{
|
||||||
|
struct vport *vport = ovs_vport_rcu(dp, out_port);
|
||||||
|
|
||||||
|
- if (likely(vport && netif_carrier_ok(vport->dev))) {
|
||||||
|
+ if (likely(vport &&
|
||||||
|
+ netif_running(vport->dev) &&
|
||||||
|
+ netif_carrier_ok(vport->dev))) {
|
||||||
|
u16 mru = OVS_CB(skb)->mru;
|
||||||
|
u32 cutlen = OVS_CB(skb)->cutlen;
|
||||||
|
|
||||||
|
--
|
||||||
|
2.39.5
|
||||||
|
|
Loading…
Reference in New Issue
Block a user