mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson
synced 2025-08-30 03:57:32 +00:00
f0e1a0643a
Implement a new scheduler class sched_ext (SCX), which allows scheduling
policies to be implemented as BPF programs to achieve the following:
1. Ease of experimentation and exploration: Enabling rapid iteration of new
scheduling policies.
2. Customization: Building application-specific schedulers which implement
policies that are not applicable to general-purpose schedulers.
3. Rapid scheduler deployments: Non-disruptive swap outs of scheduling
policies in production environments.
sched_ext leverages BPF’s struct_ops feature to define a structure which
exports function callbacks and flags to BPF programs that wish to implement
scheduling policies. The struct_ops structure exported by sched_ext is
struct sched_ext_ops, and is conceptually similar to struct sched_class. The
role of sched_ext is to map the complex sched_class callbacks to the more
simple and ergonomic struct sched_ext_ops callbacks.
For more detailed discussion on the motivations and overview, please refer
to the cover letter.
Later patches will also add several example schedulers and documentation.
This patch implements the minimum core framework to enable implementation of
BPF schedulers. Subsequent patches will gradually add functionalities
including safety guarantee mechanisms, nohz and cgroup support.
include/linux/sched/ext.h defines struct sched_ext_ops. With the comment on
top, each operation should be self-explanatory. The followings are worth
noting:
- Both "sched_ext" and its shorthand "scx" are used. If the identifier
already has "sched" in it, "ext" is used; otherwise, "scx".
- In sched_ext_ops, only .name is mandatory. Every operation is optional and
if omitted a simple but functional default behavior is provided.
- A new policy constant SCHED_EXT is added and a task can select sched_ext
by invoking sched_setscheduler(2) with the new policy constant. However,
if the BPF scheduler is not loaded, SCHED_EXT is the same as SCHED_NORMAL
and the task is scheduled by CFS. When the BPF scheduler is loaded, all
tasks which have the SCHED_EXT policy are switched to sched_ext.
- To bridge the workflow imbalance between the scheduler core and
sched_ext_ops callbacks, sched_ext uses simple FIFOs called dispatch
queues (dsq's). By default, there is one global dsq (SCX_DSQ_GLOBAL), and
one local per-CPU dsq (SCX_DSQ_LOCAL). SCX_DSQ_GLOBAL is provided for
convenience and need not be used by a scheduler that doesn't require it.
SCX_DSQ_LOCAL is the per-CPU FIFO that sched_ext pulls from when putting
the next task on the CPU. The BPF scheduler can manage an arbitrary number
of dsq's using scx_bpf_create_dsq() and scx_bpf_destroy_dsq().
- sched_ext guarantees system integrity no matter what the BPF scheduler
does. To enable this, each task's ownership is tracked through
p->scx.ops_state and all tasks are put on scx_tasks list. The disable path
can always recover and revert all tasks back to CFS. See p->scx.ops_state
and scx_tasks.
- A task is not tied to its rq while enqueued. This decouples CPU selection
from queueing and allows sharing a scheduling queue across an arbitrary
subset of CPUs. This adds some complexities as a task may need to be
bounced between rq's right before it starts executing. See
dispatch_to_local_dsq() and move_task_to_local_dsq().
- One complication that arises from the above weak association between task
and rq is that synchronizing with dequeue() gets complicated as dequeue()
may happen anytime while the task is enqueued and the dispatch path might
need to release the rq lock to transfer the task. Solving this requires a
bit of complexity. See the logic around p->scx.sticky_cpu and
p->scx.ops_qseq.
- Both enable and disable paths are a bit complicated. The enable path
switches all tasks without blocking to avoid issues which can arise from
partially switched states (e.g. the switching task itself being starved).
The disable path can't trust the BPF scheduler at all, so it also has to
guarantee forward progress without blocking. See scx_ops_enable() and
scx_ops_disable_workfn().
- When sched_ext is disabled, static_branches are used to shut down the
entry points from hot paths.
v7: - scx_ops_bypass() was incorrectly and unnecessarily trying to grab
scx_ops_enable_mutex which can lead to deadlocks in the disable path.
Fixed.
- Fixed TASK_DEAD handling bug in scx_ops_enable() path which could lead
to use-after-free.
- Consolidated per-cpu variable usages and other cleanups.
v6: - SCX_NR_ONLINE_OPS replaced with SCX_OPI_*_BEGIN/END so that multiple
groups can be expressed. Later CPU hotplug operations are put into
their own group.
- SCX_OPS_DISABLING state is replaced with the new bypass mechanism
which allows temporarily putting the system into simple FIFO
scheduling mode bypassing the BPF scheduler. In addition to the shut
down path, this will also be used to isolate the BPF scheduler across
PM events. Enabling and disabling the bypass mode requires iterating
all runnable tasks. rq->scx.runnable_list addition is moved from the
later watchdog patch.
- ops.prep_enable() is replaced with ops.init_task() and
ops.enable/disable() are now called whenever the task enters and
leaves sched_ext instead of when the task becomes schedulable on
sched_ext and stops being so. A new operation - ops.exit_task() - is
called when the task stops being schedulable on sched_ext.
- scx_bpf_dispatch() can now be called from ops.select_cpu() too. This
removes the need for communicating local dispatch decision made by
ops.select_cpu() to ops.enqueue() via per-task storage.
SCX_KF_SELECT_CPU is added to support the change.
- SCX_TASK_ENQ_LOCAL which told the BPF scheudler that
scx_select_cpu_dfl() wants the task to be dispatched to the local DSQ
was removed. Instead, scx_bpf_select_cpu_dfl() now dispatches directly
if it finds a suitable idle CPU. If such behavior is not desired,
users can use scx_bpf_select_cpu_dfl() which returns the verdict in a
bool out param.
- scx_select_cpu_dfl() was mishandling WAKE_SYNC and could end up
queueing many tasks on a local DSQ which makes tasks to execute in
order while other CPUs stay idle which made some hackbench numbers
really bad. Fixed.
- The current state of sched_ext can now be monitored through files
under /sys/sched_ext instead of /sys/kernel/debug/sched/ext. This is
to enable monitoring on kernels which don't enable debugfs.
- sched_ext wasn't telling BPF that ops.dispatch()'s @prev argument may
be NULL and a BPF scheduler which derefs the pointer without checking
could crash the kernel. Tell BPF. This is currently a bit ugly. A
better way to annotate this is expected in the future.
- scx_exit_info updated to carry pointers to message buffers instead of
embedding them directly. This decouples buffer sizes from API so that
they can be changed without breaking compatibility.
- exit_code added to scx_exit_info. This is used to indicate different
exit conditions on non-error exits and will be used to handle e.g. CPU
hotplugs.
- The patch "sched_ext: Allow BPF schedulers to switch all eligible
tasks into sched_ext" is folded in and the interface is changed so
that partial switching is indicated with a new ops flag
%SCX_OPS_SWITCH_PARTIAL. This makes scx_bpf_switch_all() unnecessasry
and in turn SCX_KF_INIT. ops.init() is now called with
SCX_KF_SLEEPABLE.
- Code reorganized so that only the parts necessary to integrate with
the rest of the kernel are in the header files.
- Changes to reflect the BPF and other kernel changes including the
addition of bpf_sched_ext_ops.cfi_stubs.
v5: - To accommodate 32bit configs, p->scx.ops_state is now atomic_long_t
instead of atomic64_t and scx_dsp_buf_ent.qseq which uses
load_acquire/store_release is now unsigned long instead of u64.
- Fix the bug where bpf_scx_btf_struct_access() was allowing write
access to arbitrary fields.
- Distinguish kfuncs which can be called from any sched_ext ops and from
anywhere. e.g. scx_bpf_pick_idle_cpu() can now be called only from
sched_ext ops.
- Rename "type" to "kind" in scx_exit_info to make it easier to use on
languages in which "type" is a reserved keyword.
- Since cff9b2332a ("kernel/sched: Modify initial boot task idle
setup"), PF_IDLE is not set on idle tasks which haven't been online
yet which made scx_task_iter_next_filtered() include those idle tasks
in iterations leading to oopses. Update scx_task_iter_next_filtered()
to directly test p->sched_class against idle_sched_class instead of
using is_idle_task() which tests PF_IDLE.
- Other updates to match upstream changes such as adding const to
set_cpumask() param and renaming check_preempt_curr() to
wakeup_preempt().
v4: - SCHED_CHANGE_BLOCK replaced with the previous
sched_deq_and_put_task()/sched_enq_and_set_tsak() pair. This is
because upstream is adaopting a different generic cleanup mechanism.
Once that lands, the code will be adapted accordingly.
- task_on_scx() used to test whether a task should be switched into SCX,
which is confusing. Renamed to task_should_scx(). task_on_scx() now
tests whether a task is currently on SCX.
- scx_has_idle_cpus is barely used anymore and replaced with direct
check on the idle cpumask.
- SCX_PICK_IDLE_CORE added and scx_pick_idle_cpu() improved to prefer
fully idle cores.
- ops.enable() now sees up-to-date p->scx.weight value.
- ttwu_queue path is disabled for tasks on SCX to avoid confusing BPF
schedulers expecting ->select_cpu() call.
- Use cpu_smt_mask() instead of topology_sibling_cpumask() like the rest
of the scheduler.
v3: - ops.set_weight() added to allow BPF schedulers to track weight changes
without polling p->scx.weight.
- move_task_to_local_dsq() was losing SCX-specific enq_flags when
enqueueing the task on the target dsq because it goes through
activate_task() which loses the upper 32bit of the flags. Carry the
flags through rq->scx.extra_enq_flags.
- scx_bpf_dispatch(), scx_bpf_pick_idle_cpu(), scx_bpf_task_running()
and scx_bpf_task_cpu() now use the new KF_RCU instead of
KF_TRUSTED_ARGS to make it easier for BPF schedulers to call them.
- The kfunc helper access control mechanism implemented through
sched_ext_entity.kf_mask is improved. Now SCX_CALL_OP*() is always
used when invoking scx_ops operations.
v2: - balance_scx_on_up() is dropped. Instead, on UP, balance_scx() is
called from put_prev_taks_scx() and pick_next_task_scx() as necessary.
To determine whether balance_scx() should be called from
put_prev_task_scx(), SCX_TASK_DEQD_FOR_SLEEP flag is added. See the
comment in put_prev_task_scx() for details.
- sched_deq_and_put_task() / sched_enq_and_set_task() sequences replaced
with SCHED_CHANGE_BLOCK().
- Unused all_dsqs list removed. This was a left-over from previous
iterations.
- p->scx.kf_mask is added to track and enforce which kfunc helpers are
allowed. Also, init/exit sequences are updated to make some kfuncs
always safe to call regardless of the current BPF scheduler state.
Combined, this should make all the kfuncs safe.
- BPF now supports sleepable struct_ops operations. Hacky workaround
removed and operations and kfunc helpers are tagged appropriately.
- BPF now supports bitmask / cpumask helpers. scx_bpf_get_idle_cpumask()
and friends are added so that BPF schedulers can use the idle masks
with the generic helpers. This replaces the hacky kfunc helpers added
by a separate patch in V1.
- CONFIG_SCHED_CLASS_EXT can no longer be enabled if SCHED_CORE is
enabled. This restriction will be removed by a later patch which adds
core-sched support.
- Add MAINTAINERS entries and other misc changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
Co-authored-by: David Vernet <dvernet@meta.com>
Acked-by: Josh Don <joshdon@google.com>
Acked-by: Hao Luo <haoluo@google.com>
Acked-by: Barret Rhoden <brho@google.com>
Cc: Andrea Righi <andrea.righi@canonical.com>
150 lines
6.2 KiB
C
150 lines
6.2 KiB
C
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
|
|
#ifndef _UAPI_LINUX_SCHED_H
|
|
#define _UAPI_LINUX_SCHED_H
|
|
|
|
#include <linux/types.h>
|
|
|
|
/*
|
|
* cloning flags:
|
|
*/
|
|
#define CSIGNAL 0x000000ff /* signal mask to be sent at exit */
|
|
#define CLONE_VM 0x00000100 /* set if VM shared between processes */
|
|
#define CLONE_FS 0x00000200 /* set if fs info shared between processes */
|
|
#define CLONE_FILES 0x00000400 /* set if open files shared between processes */
|
|
#define CLONE_SIGHAND 0x00000800 /* set if signal handlers and blocked signals shared */
|
|
#define CLONE_PIDFD 0x00001000 /* set if a pidfd should be placed in parent */
|
|
#define CLONE_PTRACE 0x00002000 /* set if we want to let tracing continue on the child too */
|
|
#define CLONE_VFORK 0x00004000 /* set if the parent wants the child to wake it up on mm_release */
|
|
#define CLONE_PARENT 0x00008000 /* set if we want to have the same parent as the cloner */
|
|
#define CLONE_THREAD 0x00010000 /* Same thread group? */
|
|
#define CLONE_NEWNS 0x00020000 /* New mount namespace group */
|
|
#define CLONE_SYSVSEM 0x00040000 /* share system V SEM_UNDO semantics */
|
|
#define CLONE_SETTLS 0x00080000 /* create a new TLS for the child */
|
|
#define CLONE_PARENT_SETTID 0x00100000 /* set the TID in the parent */
|
|
#define CLONE_CHILD_CLEARTID 0x00200000 /* clear the TID in the child */
|
|
#define CLONE_DETACHED 0x00400000 /* Unused, ignored */
|
|
#define CLONE_UNTRACED 0x00800000 /* set if the tracing process can't force CLONE_PTRACE on this clone */
|
|
#define CLONE_CHILD_SETTID 0x01000000 /* set the TID in the child */
|
|
#define CLONE_NEWCGROUP 0x02000000 /* New cgroup namespace */
|
|
#define CLONE_NEWUTS 0x04000000 /* New utsname namespace */
|
|
#define CLONE_NEWIPC 0x08000000 /* New ipc namespace */
|
|
#define CLONE_NEWUSER 0x10000000 /* New user namespace */
|
|
#define CLONE_NEWPID 0x20000000 /* New pid namespace */
|
|
#define CLONE_NEWNET 0x40000000 /* New network namespace */
|
|
#define CLONE_IO 0x80000000 /* Clone io context */
|
|
|
|
/* Flags for the clone3() syscall. */
|
|
#define CLONE_CLEAR_SIGHAND 0x100000000ULL /* Clear any signal handler and reset to SIG_DFL. */
|
|
#define CLONE_INTO_CGROUP 0x200000000ULL /* Clone into a specific cgroup given the right permissions. */
|
|
|
|
/*
|
|
* cloning flags intersect with CSIGNAL so can be used with unshare and clone3
|
|
* syscalls only:
|
|
*/
|
|
#define CLONE_NEWTIME 0x00000080 /* New time namespace */
|
|
|
|
#ifndef __ASSEMBLY__
|
|
/**
|
|
* struct clone_args - arguments for the clone3 syscall
|
|
* @flags: Flags for the new process as listed above.
|
|
* All flags are valid except for CSIGNAL and
|
|
* CLONE_DETACHED.
|
|
* @pidfd: If CLONE_PIDFD is set, a pidfd will be
|
|
* returned in this argument.
|
|
* @child_tid: If CLONE_CHILD_SETTID is set, the TID of the
|
|
* child process will be returned in the child's
|
|
* memory.
|
|
* @parent_tid: If CLONE_PARENT_SETTID is set, the TID of
|
|
* the child process will be returned in the
|
|
* parent's memory.
|
|
* @exit_signal: The exit_signal the parent process will be
|
|
* sent when the child exits.
|
|
* @stack: Specify the location of the stack for the
|
|
* child process.
|
|
* Note, @stack is expected to point to the
|
|
* lowest address. The stack direction will be
|
|
* determined by the kernel and set up
|
|
* appropriately based on @stack_size.
|
|
* @stack_size: The size of the stack for the child process.
|
|
* @tls: If CLONE_SETTLS is set, the tls descriptor
|
|
* is set to tls.
|
|
* @set_tid: Pointer to an array of type *pid_t. The size
|
|
* of the array is defined using @set_tid_size.
|
|
* This array is used to select PIDs/TIDs for
|
|
* newly created processes. The first element in
|
|
* this defines the PID in the most nested PID
|
|
* namespace. Each additional element in the array
|
|
* defines the PID in the parent PID namespace of
|
|
* the original PID namespace. If the array has
|
|
* less entries than the number of currently
|
|
* nested PID namespaces only the PIDs in the
|
|
* corresponding namespaces are set.
|
|
* @set_tid_size: This defines the size of the array referenced
|
|
* in @set_tid. This cannot be larger than the
|
|
* kernel's limit of nested PID namespaces.
|
|
* @cgroup: If CLONE_INTO_CGROUP is specified set this to
|
|
* a file descriptor for the cgroup.
|
|
*
|
|
* The structure is versioned by size and thus extensible.
|
|
* New struct members must go at the end of the struct and
|
|
* must be properly 64bit aligned.
|
|
*/
|
|
struct clone_args {
|
|
__aligned_u64 flags;
|
|
__aligned_u64 pidfd;
|
|
__aligned_u64 child_tid;
|
|
__aligned_u64 parent_tid;
|
|
__aligned_u64 exit_signal;
|
|
__aligned_u64 stack;
|
|
__aligned_u64 stack_size;
|
|
__aligned_u64 tls;
|
|
__aligned_u64 set_tid;
|
|
__aligned_u64 set_tid_size;
|
|
__aligned_u64 cgroup;
|
|
};
|
|
#endif
|
|
|
|
#define CLONE_ARGS_SIZE_VER0 64 /* sizeof first published struct */
|
|
#define CLONE_ARGS_SIZE_VER1 80 /* sizeof second published struct */
|
|
#define CLONE_ARGS_SIZE_VER2 88 /* sizeof third published struct */
|
|
|
|
/*
|
|
* Scheduling policies
|
|
*/
|
|
#define SCHED_NORMAL 0
|
|
#define SCHED_FIFO 1
|
|
#define SCHED_RR 2
|
|
#define SCHED_BATCH 3
|
|
/* SCHED_ISO: reserved but not implemented yet */
|
|
#define SCHED_IDLE 5
|
|
#define SCHED_DEADLINE 6
|
|
#define SCHED_EXT 7
|
|
|
|
/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */
|
|
#define SCHED_RESET_ON_FORK 0x40000000
|
|
|
|
/*
|
|
* For the sched_{set,get}attr() calls
|
|
*/
|
|
#define SCHED_FLAG_RESET_ON_FORK 0x01
|
|
#define SCHED_FLAG_RECLAIM 0x02
|
|
#define SCHED_FLAG_DL_OVERRUN 0x04
|
|
#define SCHED_FLAG_KEEP_POLICY 0x08
|
|
#define SCHED_FLAG_KEEP_PARAMS 0x10
|
|
#define SCHED_FLAG_UTIL_CLAMP_MIN 0x20
|
|
#define SCHED_FLAG_UTIL_CLAMP_MAX 0x40
|
|
|
|
#define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \
|
|
SCHED_FLAG_KEEP_PARAMS)
|
|
|
|
#define SCHED_FLAG_UTIL_CLAMP (SCHED_FLAG_UTIL_CLAMP_MIN | \
|
|
SCHED_FLAG_UTIL_CLAMP_MAX)
|
|
|
|
#define SCHED_FLAG_ALL (SCHED_FLAG_RESET_ON_FORK | \
|
|
SCHED_FLAG_RECLAIM | \
|
|
SCHED_FLAG_DL_OVERRUN | \
|
|
SCHED_FLAG_KEEP_ALL | \
|
|
SCHED_FLAG_UTIL_CLAMP)
|
|
|
|
#endif /* _UAPI_LINUX_SCHED_H */
|