mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson
synced 2025-08-29 11:36:31 +00:00
ee62ce7a1d
When enabling DMA mapping in page_pool, pages are kept DMA mapped until
they are released from the pool, to avoid the overhead of re-mapping the
pages every time they are used. This causes resource leaks and/or
crashes when there are pages still outstanding while the device is torn
down, because page_pool will attempt an unmap through a non-existent DMA
device on the subsequent page return.
To fix this, implement a simple tracking of outstanding DMA-mapped pages
in page pool using an xarray. This was first suggested by Mina[0], and
turns out to be fairly straight forward: We simply store pointers to
pages directly in the xarray with xa_alloc() when they are first DMA
mapped, and remove them from the array on unmap. Then, when a page pool
is torn down, it can simply walk the xarray and unmap all pages still
present there before returning, which also allows us to get rid of the
get/put_device() calls in page_pool. Using xa_cmpxchg(), no additional
synchronisation is needed, as a page will only ever be unmapped once.
To avoid having to walk the entire xarray on unmap to find the page
reference, we stash the ID assigned by xa_alloc() into the page
structure itself, using the upper bits of the pp_magic field. This
requires a couple of defines to avoid conflicting with the
POINTER_POISON_DELTA define, but this is all evaluated at compile-time,
so does not affect run-time performance. The bitmap calculations in this
patch gives the following number of bits for different architectures:
- 23 bits on 32-bit architectures
- 21 bits on PPC64 (because of the definition of ILLEGAL_POINTER_VALUE)
- 32 bits on other 64-bit architectures
Stashing a value into the unused bits of pp_magic does have the effect
that it can make the value stored there lie outside the unmappable
range (as governed by the mmap_min_addr sysctl), for architectures that
don't define ILLEGAL_POINTER_VALUE. This means that if one of the
pointers that is aliased to the pp_magic field (such as page->lru.next)
is dereferenced while the page is owned by page_pool, that could lead to
a dereference into userspace, which is a security concern. The risk of
this is mitigated by the fact that (a) we always clear pp_magic before
releasing a page from page_pool, and (b) this would need a
use-after-free bug for struct page, which can have many other risks
since page->lru.next is used as a generic list pointer in multiple
places in the kernel. As such, with this patch we take the position that
this risk is negligible in practice. For more discussion, see[1].
Since all the tracking added in this patch is performed on DMA
map/unmap, no additional code is needed in the fast path, meaning the
performance overhead of this tracking is negligible there. A
micro-benchmark shows that the total overhead of the tracking itself is
about 400 ns (39 cycles(tsc) 395.218 ns; sum for both map and unmap[2]).
Since this cost is only paid on DMA map and unmap, it seems like an
acceptable cost to fix the late unmap issue. Further optimisation can
narrow the cases where this cost is paid (for instance by eliding the
tracking when DMA map/unmap is a no-op).
The extra memory needed to track the pages is neatly encapsulated inside
xarray, which uses the 'struct xa_node' structure to track items. This
structure is 576 bytes long, with slots for 64 items, meaning that a
full node occurs only 9 bytes of overhead per slot it tracks (in
practice, it probably won't be this efficient, but in any case it should
be an acceptable overhead).
[0] https://lore.kernel.org/all/CAHS8izPg7B5DwKfSuzz-iOop_YRbk3Sd6Y4rX7KBG9DcVJcyWg@mail.gmail.com/
[1] https://lore.kernel.org/r/20250320023202.GA25514@openwall.com
[2] https://lore.kernel.org/r/ae07144c-9295-4c9d-a400-153bb689fe9e@huawei.com
Reported-by: Yonglong Liu <liuyonglong@huawei.com>
Closes: https://lore.kernel.org/r/8743264a-9700-4227-a556-5f931c720211@huawei.com
Fixes: ff7d6b27f8 ("page_pool: refurbish version of page_pool code")
Suggested-by: Mina Almasry <almasrymina@google.com>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Reviewed-by: Jesper Dangaard Brouer <hawk@kernel.org>
Tested-by: Jesper Dangaard Brouer <hawk@kernel.org>
Tested-by: Qiuling Ren <qren@redhat.com>
Tested-by: Yuying Ma <yuma@redhat.com>
Tested-by: Yonglong Liu <liuyonglong@huawei.com>
Acked-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Link: https://patch.msgid.link/20250409-page-pool-track-dma-v9-2-6a9ef2e0cba8@redhat.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
94 lines
2.8 KiB
C
94 lines
2.8 KiB
C
/* SPDX-License-Identifier: GPL-2.0 */
|
|
#ifndef _LINUX_POISON_H
|
|
#define _LINUX_POISON_H
|
|
|
|
/********** include/linux/list.h **********/
|
|
|
|
/*
|
|
* Architectures might want to move the poison pointer offset
|
|
* into some well-recognized area such as 0xdead000000000000,
|
|
* that is also not mappable by user-space exploits:
|
|
*/
|
|
#ifdef CONFIG_ILLEGAL_POINTER_VALUE
|
|
# define POISON_POINTER_DELTA _AC(CONFIG_ILLEGAL_POINTER_VALUE, UL)
|
|
#else
|
|
# define POISON_POINTER_DELTA 0
|
|
#endif
|
|
|
|
/*
|
|
* These are non-NULL pointers that will result in page faults
|
|
* under normal circumstances, used to verify that nobody uses
|
|
* non-initialized list entries.
|
|
*/
|
|
#define LIST_POISON1 ((void *) 0x100 + POISON_POINTER_DELTA)
|
|
#define LIST_POISON2 ((void *) 0x122 + POISON_POINTER_DELTA)
|
|
|
|
/********** include/linux/timer.h **********/
|
|
#define TIMER_ENTRY_STATIC ((void *) 0x300 + POISON_POINTER_DELTA)
|
|
|
|
/********** mm/page_poison.c **********/
|
|
#define PAGE_POISON 0xaa
|
|
|
|
/********** mm/page_alloc.c ************/
|
|
|
|
#define TAIL_MAPPING ((void *) 0x400 + POISON_POINTER_DELTA)
|
|
|
|
/********** mm/slab.c **********/
|
|
/*
|
|
* Magic nums for obj red zoning.
|
|
* Placed in the first word before and the first word after an obj.
|
|
*/
|
|
#define SLUB_RED_INACTIVE 0xbb /* when obj is inactive */
|
|
#define SLUB_RED_ACTIVE 0xcc /* when obj is active */
|
|
|
|
/* ...and for poisoning */
|
|
#define POISON_INUSE 0x5a /* for use-uninitialised poisoning */
|
|
#define POISON_FREE 0x6b /* for use-after-free poisoning */
|
|
#define POISON_END 0xa5 /* end-byte of poisoning */
|
|
|
|
/********** arch/$ARCH/mm/init.c **********/
|
|
#define POISON_FREE_INITMEM 0xcc
|
|
|
|
/********** fs/jbd/journal.c **********/
|
|
#define JBD_POISON_FREE 0x5b
|
|
#define JBD2_POISON_FREE 0x5c
|
|
|
|
/********** drivers/base/dmapool.c **********/
|
|
#define POOL_POISON_FREED 0xa7 /* !inuse */
|
|
#define POOL_POISON_ALLOCATED 0xa9 /* !initted */
|
|
|
|
/********** drivers/atm/ **********/
|
|
#define ATM_POISON_FREE 0x12
|
|
#define ATM_POISON 0xdeadbeef
|
|
|
|
/********** kernel/mutexes **********/
|
|
#define MUTEX_DEBUG_INIT 0x11
|
|
#define MUTEX_DEBUG_FREE 0x22
|
|
#define MUTEX_POISON_WW_CTX ((void *) 0x500 + POISON_POINTER_DELTA)
|
|
|
|
/********** security/ **********/
|
|
#define KEY_DESTROY 0xbd
|
|
|
|
/********** net/core/page_pool.c **********/
|
|
/*
|
|
* page_pool uses additional free bits within this value to store data, see the
|
|
* definition of PP_DMA_INDEX_MASK in mm.h
|
|
*/
|
|
#define PP_SIGNATURE (0x40 + POISON_POINTER_DELTA)
|
|
|
|
/********** net/core/skbuff.c **********/
|
|
#define SKB_LIST_POISON_NEXT ((void *)(0x800 + POISON_POINTER_DELTA))
|
|
/********** net/ **********/
|
|
#define NET_PTR_POISON ((void *)(0x801 + POISON_POINTER_DELTA))
|
|
|
|
/********** kernel/bpf/ **********/
|
|
#define BPF_PTR_POISON ((void *)(0xeB9FUL + POISON_POINTER_DELTA))
|
|
|
|
/********** VFS **********/
|
|
#define VFS_PTR_POISON ((void *)(0xF5 + POISON_POINTER_DELTA))
|
|
|
|
/********** lib/stackdepot.c **********/
|
|
#define STACK_DEPOT_POISON ((void *)(0xD390 + POISON_POINTER_DELTA))
|
|
|
|
#endif
|