Shasta: Interactive Reporting At Scale

We describe Shasta, a middleware system built at Google to support
interactive reporting in complex user-facing applications related to
Google’s Internet advertising business. Shasta targets applications
with challenging requirements: First, user query latencies must
be low. Second, underlying transactional data stores have complex
“read-unfriendly” schemas, placing significant transformation
logic between stored data and the read-only views that Shasta exposes
to its clients. This transformation logic must be expressed
in a way that scales to large and agile engineering teams. Finally,
Shasta targets applications with strong data freshness requirements,
making it challenging to precompute query results using common
techniques such as ETL pipelines or materialized views. Instead,
online queries must go all the way from primary storage to userfacing
views, resulting in complex queries joining 50 or more tables.
Designed as a layer on top of Google’s F1 RDBMS and Mesa
data warehouse, Shasta combines language and system techniques
to meet these requirements. To help with expressing complex view
specifications, we developed a query language called RVL, with
support for modularized view templates that can be dynamically
compiled into SQL. To execute these SQL queries with low latency
at scale, we leveraged and extended F1’s distributed query engine
with facilities such as safe execution of C++ and Java UDFs. To
reduce latency and increase read parallelism, we extended F1 storage
with a distributed read-only in-memory cache. The system we
describe is in production at Google, powering critical applications
used by advertisers and internal sales teams. Shasta has signifi-
cantly improved system scalability and software engineering effi-
ciency compared to the middleware solutions it replaced.

Source: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45394.pdf

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s