FROMLIST: sched: Consolidate cpufreq updates
Improve the interaction with cpufreq governors by making the
cpufreq_update_util() calls more intentional.
At the moment we send them when load is updated for CFS, bandwidth for
DL and at enqueue/dequeue for RT. But this can lead to too many updates
sent in a short period of time and potentially be ignored at a critical
moment due to the rate_limit_us in schedutil.
For example, simultaneous task enqueue on the CPU where 2nd task is
bigger and requires higher freq. The trigger to cpufreq_update_util() by
the first task will lead to dropping the 2nd request until tick. Or
another CPU in the same policy triggers a freq update shortly after.
Updates at enqueue for RT are not strictly required. Though they do help
to reduce the delay for switching the frequency and the potential
observation of lower frequency during this delay. But current logic
doesn't intentionally (at least to my understanding) try to speed up the
request.
To help reduce the amount of cpufreq updates and make them more
purposeful, consolidate them into these locations:
1. context_switch()
2. task_tick_fair()
3. update_blocked_averages()
4. on syscall that changes policy or uclamp values
The update at context switch should help guarantee that DL and RT get
the right frequency straightaway when they're RUNNING. As mentioned
though the update will happen slightly after enqueue_task(); though in
an ideal world these tasks should be RUNNING ASAP and this additional
delay should be negligible. For fair tasks we need to make sure we send
a single update for every decay for the root cfs_rq. Any changes to the
rq will be deferred until the next task is ready to run, or we hit TICK.
But we are guaranteed the task is running at a level that meets its
requirements after enqueue.
To guarantee RT and DL tasks updates are never missed, we add a new
SCHED_CPUFREQ_FORCE_UPDATE to ignore the rate_limit_us. If we are
already running at the right freq, the governor will end up doing
nothing, but we eliminate the risk of the task ending up accidentally
running at the wrong freq due to rate_limit_us.
Similarly for iowait boost, we ignore rate limits. We also handle a case
of a boost reset prematurely by adding a guard in sugov_iowait_apply()
to reduce the boost after 1ms which seems iowait boost mechanism relied
on rate_limit_us and cfs_rq.decay preventing any updates to happen soon
after iowait boost.
The new SCHED_CPUFREQ_FORCE_UPDATE should not impact the rate limit
time stamps otherwise we can end up delaying updates for normal
requests.
As a simple optimization, we avoid sending cpufreq updates when
switching from RT to another RT as RT tasks run at max freq by default.
If CONFIG_UCLAMP_TASK is enabled, we can do a simple check to see if
uclamp_min is different to avoid unnecessary cpufreq update as most RT
tasks are likely to be running at the same performance level, so we can
avoid unnecessary overhead of forced updates when there's nothing to do.
We also ensure to ignore cpufreq udpates for sugov workers at context
switch. It doesn't make sense for the kworker that applies the frequency
update (which is a DL task) to trigger a frequency update itself.
The update at task_tick_fair will guarantee that the governor will
follow any updates to load for tasks/CPU or due to new enqueues/dequeues
to the rq. Since DL and RT always run at constant frequencies and have
no load tracking, this is only required for fair tasks.
The update at update_blocked_averages() will ensure we decay frequency
as the CPU becomes idle for long enough.
If the currently running task changes its policy or uclamp values, we
ensure we follow up with cpufreq update to ensure we follow up with any
potential new perf requirements based on the new change.
Results of
taskset 1 perf stat --repeat 10 -e cycles,instructions,task-clock perf bench sched pipe
on AMD 3900X to verify any potential overhead because of the addition at
context switch against v6.8.7 stable kernel
v6.8.7: schedutil:
------------------
Performance counter stats for 'perf bench sched pipe' (10 runs):
850,276,689 cycles:u # 0.078 GHz ( +- 0.88% )
82,724,245 instructions:u # 0.10 insn per cycle ( +- 0.00% )
10,881.41 msec task-clock:u # 0.995 CPUs utilized ( +- 0.12% )
10.9377 +- 0.0135 seconds time elapsed ( +- 0.12% )
v6.8.7: performance:
--------------------
Performance counter stats for 'perf bench sched pipe' (10 runs):
874,154,415 cycles:u # 0.080 GHz ( +- 0.78% )
82,724,420 instructions:u # 0.10 insn per cycle ( +- 0.00% )
10,916.47 msec task-clock:u # 0.999 CPUs utilized ( +- 0.09% )
10.9308 +- 0.0100 seconds time elapsed ( +- 0.09% )
v6.8.7+patch: schedutil:
------------------------
Performance counter stats for 'perf bench sched pipe' (10 runs):
816,938,281 cycles:u # 0.075 GHz ( +- 0.84% )
82,724,163 instructions:u # 0.10 insn per cycle ( +- 0.00% )
10,907.62 msec task-clock:u # 1.004 CPUs utilized ( +- 0.11% )
10.8627 +- 0.0121 seconds time elapsed ( +- 0.11% )
v6.8.7+patch: performance:
--------------------------
Performance counter stats for 'perf bench sched pipe' (10 runs):
814,038,416 cycles:u # 0.074 GHz ( +- 1.21% )
82,724,356 instructions:u # 0.10 insn per cycle ( +- 0.00% )
10,886.69 msec task-clock:u # 0.996 CPUs utilized ( +- 0.17% )
10.9298 +- 0.0181 seconds time elapsed ( +- 0.17% )
Note worthy that we still have the following race condition on systems
that have shared policy:
* CPUs with shared policy can end up sending simultaneous cpufreq
updates requests where the 2nd one will be unlucky and get blocked by
the rate_limit_us (schedutil).
We can potentially address this limitation later, but it is out of the
scope of this patch.
Bug: 340190651
Link: https://lore.kernel.org/lkml/20240516204802.846520-1-qyousef@layalina.io/
Signed-off-by: 
Qais Yousef <qyousef@layalina.io>
[Fix trivial conflicts in sugov functions due to reworks done there after 6.6]
Signed-off-by: 
Qais Yousef <qyousef@google.com>
Change-Id: Iadc7128b33dbf78e66b83d04018d2162b109f3dc
Loading