BACKPORT: rcu: Make call_rcu() lazy to save power
Implement timer-based RCU callback batching (also known as lazy callbacks). With this we save about 5-10% of power consumed due to RCU requests that happen when system is lightly loaded or idle. By default, all async callbacks (queued via call_rcu) are marked lazy. An alternate API call_rcu_hurry() is provided for the few users, for example synchronize_rcu(), that need the old behavior. The batch is flushed whenever a certain amount of time has passed, or the batch on a particular CPU grows too big. Also memory pressure will flush it in a future patch. To handle several corner cases automagically (such as rcu_barrier() and hotplug), we re-use bypass lists which were originally introduced to address lock contention, to handle lazy CBs as well. The bypass list length has the lazy CB length included in it. A separate lazy CB length counter is also introduced to keep track of the number of lazy CBs. [ paulmck: Fix formatting of inline call_rcu_lazy() definition. ] [ paulmck: Apply Zqiang feedback. ] [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] [ joelaf: Small changes for 5.15 backport. ] Suggested-by:Paul McKenney <paulmck@kernel.org> Acked-by:
Frederic Weisbecker <frederic@kernel.org> Signed-off-by:
Joel Fernandes (Google) <joel@joelfernandes.org> Signed-off-by:
Vineeth Pillai <vineethrp@google.com> Commit-Queue: Joel Fernandes <joelaf@google.com> Tested-by:
Joel Fernandes <joelaf@google.com> (cherry picked from commit b30e520b9da88a5de115ed5b2c1b2aa89de9e214) [Cherry picked from chromeos-5.15 tree. Minor tweaks to commit message to match Android style] Signed-off-by:
Qais Yousef <qyousef@google.com>
Loading