Scaling traffic configurations

A sufficiently complex configuration can rot much like a codebase does. How can this happen, and how can you approach the process of fixing it? This talk explores the evolution of Facebook’s traffic configuration over the years.

Design patterns for container-based distributed systems

In the late 1980s and early 1990s, object-oriented programming
revolutionized software development, popularizing
the approach of building of applications as collections
of modular components. Today we are seeing
a similar revolution in distributed system development,
with the increasing popularity of microservice architectures
built from containerized software components.
Containers [15] [22] [1] [2] are particularly well-suited
as the fundamental “object” in distributed systems by
virtue of the walls they erect at the container boundary.
As this architectural style matures, we are seeing the
emergence of design patterns, much as we did for objectoriented
programs, and for the same reason – thinking in
terms of objects (or containers) abstracts away the lowlevel
details of code, eventually revealing higher-level
patterns that are common to a variety of applications and
This paper describes three types of design patterns
that we have observed emerging in container-based distributed
systems: single-container patterns for container
management, single-node patterns of closely cooperating
containers, and multi-node patterns for distributed
algorithms. Like object-oriented patterns before them,
these patterns for distributed computation encode best
practices, simplify development, and make the systems
where they are used more reliable.


Consistent Updates for Software-Defined Networks: Change You Can Believe In!

Configuration changes are a common source of instability in
networks, leading to broken connectivity, forwarding loops,
and access control violations. Even when the initial and final
states of the network are correct, the update process often
steps through intermediate states with incorrect behaviors.
These problems have been recognized in the context of specific
protocols, leading to a number of point solutions. However,
a piecemeal attack on this fundamental problem, while
pragmatic in the short term, is unlikely to lead to significant
long-term progress.
Software-Defined Networking (SDN) provides an exciting
opportunity to do better. Because SDN is a clean-slate
platform, we can build general, reusable abstractions for network
updates that come with strong semantic guarantees.
We believe SDN desperately needs such abstractions to make
programs simpler to design, more reliable, and easier to validate
using automated tools. Moreover, we believe these abstractions
should be provided by a runtime system, shielding
the programmer from these concerns. We propose two simple,
canonical, and effective update abstractions, and present
implementation mechanisms. We also show how to integrate
them with a network programming language, and discuss potential
applications to program verification.


Profiling a warehouse-scale computer

With the increasing prevalence of warehouse-scale (WSC)
and cloud computing, understanding the interactions of server
applications with the underlying microarchitecture becomes
ever more important in order to extract maximum performance
out of server hardware. To aid such understanding, this paper
presents a detailed microarchitectural analysis of live datacenter
jobs, measured on more than 20,000 Google machines
over a three year period, and comprising thousands of different
We first find that WSC workloads are extremely diverse,
breeding the need for architectures that can tolerate application
variability without performance loss. However, some
patterns emerge, offering opportunities for co-optimization
of hardware and software. For example, we identify common
building blocks in the lower levels of the software stack.
This “datacenter tax” can comprise nearly 30% of cycles
across jobs running in the fleet, which makes its constituents
prime candidates for hardware specialization in future server
systems-on-chips. We also uncover opportunities for classic
microarchitectural optimizations for server processors, especially
in the cache hierarchy. Typical workloads place signifi-
cant stress on instruction caches and prefer memory latency
over bandwidth. They also stall cores often, but compute heavily
in bursts. These observations motivate several interesting
directions for future warehouse-scale computers