Debugging high throughput, low-latency C/C++ systems in production is hard. At Google we developed XRay, a function call tracing system that allows Google engineers to get accurate function call traces with negligible overhead when off and moderate overhead when on, suitable for services deployed in production. XRay enables efficient function call entry/exit logging with high accuracy timestamps, and can be dynamically enabled and disabled. This white paper describes the XRay tracing system and its implementation. It also describes future plans with open sourcing XRay and engaging open source communities.
GOOGLE-WIDE PROFILING (GWP), A CONTINUOUS PROFILING INFRASTRUCTURE FOR
DATA CENTERS, PROVIDES PERFORMANCE INSIGHTS FOR CLOUD APPLICATIONS. WITH NEGLIGIBLE OVERHEAD, GWP PROVIDES STABLE, ACCURATE PROFILES AND A
DATACENTER-SCALE TOOL FOR TRADITIONAL PERFORMANCE ANALYSES. FURTHERMORE, GWP INTRODUCES NOVEL APPLICATIONS OF ITS PROFILES, SUCH AS APPLICATION PLATFORM AFFINITY MEASUREMENTS AND IDENTIFICATION OF PLATFORM-SPECIFIC, MICROARCHITECTURAL PECULIARITIES.
Dynamic program analysis and testing tools typically require inserting extra instrumentation code into the program to test. The inserted instrumentation then gathers data about the program execution and hands it off to the analysis algorithm. Various analysis algorithms can be used to perform CPU profiling, processor cache simulation, memory error detection, data race detection, etc. Usually the instrumentation is done either at run time or atcompile time – called dynamic instrumentation and compiler instrumentation, respectively. However, each of these methods has to make a compromise between performance and versatil-ity when used in industry software development. This paper presents a combined approach to instrumentationwhich takes the best of the two worlds – the low run-time overhead and unique features of compile-time instrumentation and the flexibility of dynamic instrumentation. Wepresent modifications of two testing tools that benefit from thisapproach: AddressSanitizer and MemorySanitizer. We propose benchmarks to compare different instrumentation frameworks in conditions specific to hybrid instrumenta-tion. We discuss the changes we made to one of the state-of-the-art instrumentation frameworks to significantly improve the performance of hybrid tools.
Memory access bugs, including buffer overflows and
uses of freed heap memory, remain a serious problem for
programming languages like C and C++. Many memory
error detectors exist, but most of them are either slow or
detect a limited set of bugs, or both.
This paper presents AddressSanitizer, a new memory
error detector. Our tool finds out-of-bounds accesses to
heap, stack, and global objects, as well as use-after-free
bugs. It employs a specialized memory allocator and
code instrumentation that is simple enough to be implemented
in any compiler, binary translation system, or
even in hardware.
AddressSanitizer achieves efficiency without sacrificing
comprehensiveness. Its average slowdown is just
73% yet it accurately detects bugs at the point of occurrence.
It has found over 300 previously unknown bugs in
the Chromium browser and many bugs in other software.