Shark Week: the Hunt for Speed, Vol. 1

Dec. 13, 2011

Speed is always in fashion. At least, that was the mantra last week at BrightTag.

Periodically, our engineering team takes a break from their regular diet of plankton and Java to focus exclusively on improving the performance of the BrightTag ONE platform. Fueled by a steady stream of prey food that magically appeared daily (thank you Lisa, Matt and Andrew), the team focused on making our already speedy technology even speedier. Below is a summary of Shark Week highlights and contributions from the various contributors:

  • Reduced page delivery time for slowest pages by 50-75%
  • Added internal tools to streamline testing, troubleshooting and monitoring across various parts of the platform and environment

Worked with the “Balding Eagles” group to improve the speed of data through the UI. Set up new tools to allow us to do automated UI/acceptance/behavior testing in-browser and identify bugs earlier. Working on refactoring and improving the appearance of the new graph display tool which will make future improvements easier to add.

Added instrumentation to our statistic engine which will eventually be published to our internal graphing tool. Also added integration tests to cover more complicated cases of our tag-serving functionality.

Eric, Ryan and Cody

Added detailed monitoring and real-time graphing capability to server-direct tags to improve our ability to troubleshoot issues quickly. Worked with the “Balding Eagles” group to optimize the dashboard load for our largest sites. This was accomplished by reducing the amount of data our metrics system returns for queries. Modifications were made to enable configurable “roll ups” of raw data into fewer buckets each covering a longer time. This will also allow us to more easily fine-tune our metrics system and dashboard performance over time. Also increased the throughput of server-direct tags by making our tag fires asynchronous.

Enhanced our monitoring API to give us better visibility into latency and errors within our system. Worked on a project to monitor the process of updating site configurations in the UI and the distribution of those changes across the U.S. and Europe. Also worked on pollers that enable charts to update in real time with metrics from our services.

Gus, Lead Balding Eagle
The “Balding Eagles” group set an ambitious goal to return all stats requests to the browser in less than one second by addressing the issue of latency via concurrent connections and a reduced data size. We focused on optimizing the number of tag requests that are distributed across our data centers. Data requests for our high-volume clients were further improved through the use of concurrent network connections. To support additional traffic from our growing list of clients, we also made adjustments to the amount of time a leaf node spent waiting for data to be retrieved from our persistence engines. Finally, we identified more efficient ways to display data in the UI to significantly reduce overhead.

Added Graphite to the BrightTag cauldron and demonstrated how real-time collection of metrics will help monitor individual application performance as well as the analysis of server load across our distributed systems architecture. Also worked with Alex to implement a graph aggregator for easier correlation of events.

Tested several different machine and JVM configurations to identify the best performance in terms of throughput, latency and footprint. Worked on a framework for getting data from our applications into an internal monitoring tool.

Bitey Charles

Added an asset manager to control caching and merging of our CSS and JavaScript resources. This greatly reduces the number of files and their size that must be called within our UI, improving performance. The asset manager will also reduce the occurrences of old cached files when a new release is deployed.

Optimized how exceptions are handled and load-tested the stats collector. Also added functionality to the core platform which will speed the delivery of data between client and partner sites.

Andrew and Charles both made significant contributions to last week’s shark-related activities. However, the nature of their speed-enhancing projects are confidential. And we’re not about to argue with anyone baring such sharp teeth.

NOTE: No fish were harmed during BrightTag’s inaugural Shark Week.

Read more on ReadWrite Hack, “Tales of Extreme Coding: Shark Week at BrightTag

Learn more about BrightTag ONE, our cloud-based integration platform.