Google uses eBPF schedulers to boost RPC performance

Google has outlined how it uses eBPF to run custom CPU scheduling policies in its infrastructure, delivering roughly a 5% queries-per-second improvement in some core RPC serving stacks.

Standard kernel schedulers were not well suited to the specific behaviour of Google's applications and hardware, prompting engineers to develop pluggable schedulers first through ghOSt and now through the sched_ext framework.

The work offers a view into how one of the world's largest infrastructure operators is pushing eBPF beyond its established roles in networking and observability into core kernel functions.

According to the case study, Google wanted a way to tailor scheduling policy without modifying the base kernel for every change. It also needed a method that would let engineers test and adjust scheduling rules on production systems without causing kernel panics or requiring reboots.

That led it to run application-specific scheduling policies inside the kernel using eBPF. In practice, this lets Google change how tasks are prioritised and placed on CPUs while keeping kernel mechanisms separate from policy decisions.

Scheduler changes

Google's design centres on three main techniques. The first uses eBPF maps to share task state between userspace and the kernel, allowing the system to adjust thread priorities quickly as new RPC requests are assigned to worker threads.

The second groups threads that share an address space into specific hardware cache domains, including AMD Core Complex boundaries, to improve cache locality. These assignments can be tuned to balance memory locality against scheduling latency.

The third shifts runqueue management from a strictly per-CPU model to groups of CPUs. Google said this coarser granularity improved work conservation and fairness across multi-core systems.

The reported result was about a 5% gain in queries per second for some core RPC serving stacks, along with double-digit throughput improvements for memory-intensive workloads.

Operational impact

Alongside the performance gains, Google framed the changes as an operational shift in how kernel behaviour can be tested. Instead of altering the base kernel directly, engineers can load and remove scheduling logic dynamically.

This has made it possible to test and refine execution policies on live production workloads with less operational risk, according to Google. It argued this matters because userspace daemons cannot make the same microsecond-level adjustments and offer only limited control over scheduler behaviour.

The account also explains why Google chose eBPF for the work. In-kernel execution provides the speed needed for real-time scheduling changes, while the eBPF verifier offers safeguards intended to prevent memory safety errors, infinite loops and crashes.

Beyond scheduling

Google is also applying the same separation of policy and mechanism to other parts of the kernel. It is exploring eBPF-driven memory management policies, including an in-kernel out-of-memory framework that would let eBPF trigger out-of-memory actions and choose alternative victims.

It is also evaluating a proposal known as cache_ext for per-application page cache eviction control. That would extend eBPF from CPU scheduling into another area of low-level system behaviour that can affect efficiency across large fleets.

More broadly, eBPF is being used not just for observability or networking but also to alter kernel policy in production settings. For operators with large, varied workloads, that could offer a way to create more specialised system behaviour without maintaining extensive kernel forks.

Google said the built-in scheduler remains a general-purpose tool designed to handle most workloads adequately, but its own infrastructure needed more specific tuning. Custom eBPF schedulers, it reported, gave it a way to match scheduling policy more closely to application demands and hardware characteristics.