Linux 4.x performance: Using BPF superpowers

Brendan Gregg from Netflix kicked off our technical talks with an in-depth presentation on the power of using BPF to analyze performance on Linux systems. The extended Berkeley Packet Filter is a relatively new profiling tool in the performance engineer’s toolbox that lets analysts run extremely efficient profiling code in a VM in the kernel. Brendan showed us how to write a BPF program, examples of some useful metrics, and a powerful way to visualize results using Flamegraphs. In particular, he demonstrated how to measure how long threads were blocked and how the threads were ultimately woken up. By following a chain of wakeup events across threads, Brendan showed how BPF and Flamegraphs could be used to root-cause the source of blocked CPU threads through user and kernel code, often all the way down to the metal.

Choose the Red Pill and the Blue Pill

In the movie “The Matrix,” our hero Neo must choose between
taking the Blue Pill and continuing to live in an online, synthesized
fantasy world, or taking the Red Pill and joining the real
world. The fantasy world appears to those living in it to be full of
flowers and trees and big steak dinners, but unknown to them
contains malicious Agents who can alter any portion of the world
to suit their needs. The real world, in turn, while real, has no visible
sun, and the people have only gray mush for food.
Authorization and authentication of online transactions across a
network requires a trusted path between the user and the server.
We posit that those who attempt to solve this problem by creating
the trusted path on the general-purpose operating system have
taken the Blue Pill and are living in a fantasy world. One simply
cannot properly secure a general-purpose operating system.
Solving the problem by taking the Red Pill and completely replacing
currently used operating systems with ones that we can properly
secure does not seem palatable. We suggest a solution that
involves taking both the Blue Pill and the Red Pill: providing the
trusted path by means of a separate device with a secure operating
system, used in tandem with the existing general purpose
operating system. Most user interaction occurs on the untrusted
system, with the secure device only being used to finalise
We believe that the technology required for such a device is readily
available. Obviously our idea is not a completely novel idea;
prior work in the area has had a similar goal. However, most of
those attempts have not properly addressed the requirements for
the trusted system, generally preferring to use existing generalpurpose
systems even when on a “dedicated device.” [Balfanz
1999] [Kingpin 2001] Others have a very limited scope of use.
[Blakely 2004]