Table of Contents
Nielsen is the leading global measurement and data analytics tool, innovating in its industry for over 90 years.
Headquartered in New York, Nielsen has operations in over 100 countries and covers more than 90% of the world’s population. The Nielsen Marketing Cloud combines Nielsen’s data analytics, media planning, marketing activation and data management platform capabilities in a single cloud platform. This enables clients to obtain more accurate, validated data at scale. Nielsen Marketing Cloud uses Hosted Graphite to understand what’s going on inside their systems.
The technical problem
Itai and the twenty-member Big Data group collect, process, store and analyze all the data for Nielsen Marketing Cloud.
As they offer both data-as-a-service (DaaS) and software-as-a-service (SaaS) for the AdTech industry, for Itai, that means data, and lots of it.
The team monitors both system metrics (e.g., memory usage) and application metrics (e.g., the number of records per processing cycle). However, as the company expanded, the limitations of its tech stack became more apparent, and it began updating its tools. As Itai puts it, “We started a process of transforming our entire data tech stack to one that would be able to support our growing business needs. We were looking for better tools and frameworks to replace our old tools.” After deciding on Apache Spark and Druid, the team set out to find a new monitoring tool. In addition to seamless scaling, they needed one that would:
- Ensure cost control and predictability: The monitoring tool needed to have easy-to-understand, transparent pricing, to keep costs down and guarantee reliable billing.
- Take minimal effort to get up and running: The team didn’t have time to opt for a tool with a tricky set up. They needed to continue monitoring their key metrics and KPIs as quickly as possible, so they required a tool that, according to Itai, “took little effort to set up and use...we wanted to create and view new dashboards with ease.”
- Integrate with other tools: The Big Data group uses a wide variety of services, so the new monitoring solution needed to integrate easily with their current tech stack, as well as any new tools they might use down the line.
- Require little ongoing maintenance: The team are always transforming their stack to support growing business needs. As such, they needed a tool that could scale with them without requiring a lot of precious Engineer and Developer hours for upkeep and maintenance.
- Include reliable alerting: Itai’s group needed an alerting feature to get notifications when their systems go down or behave unexpectedly to minimize downtime, reduce customer impact as efficiently as possible and keep services operational 24/7.
We wanted a monitoring solution that could easily integrate with our new big data tools.
The solution
While Hosted Graphite met all the team's requirements, Itai and the Big Data group had one major concern: spiralling costs.
We work with various vendors, and your support is the best hands down. It’s Excellent!
They planned to integrate one of the new big data tools they were using, Apache Spark, to send application metrics. However, the team was concerned that the metrics from Spark might drive up costs. Each time a Spark application starts, it generates an ID to be used as the application identifier so that a typical metric name would be:
application_<APPLICATION_ID>.driver.<APPLICATION_NAME>.<METRIC_NAME>
That way, if you have a batch application that runs every hour, it would send a metric: <METRIC_NAME> with the current <APPLICATION_ID>, so the team would end up with multiple metrics in Hosted Graphite, which are essentially the same metric. “Lucky for us, Hosted Graphite were able to solve this problem,” says Itai. The issue was fixed by removing the <APPLICATION_ID> part from the metric name coming into Graphite, such that,
application_<APPLICATION_ID>.driver.<APPLICATION_NAME>.<METRIC_NAME>
has become,
application.driver.<APPLICATION_NAME>.<METRIC_NAME>
This straightforward solution was implemented by Hosted Graphite’s technical support, without requiring Itai and the team to change anything in their code or configuration. This keeps their metric count under control, which puts the team’s budgetary concerns to rest: “Hosted Graphite charges per metric, so it’s very easy to understand how to keep costs under control.” In addition, according to Itai, “when we saw how easy it was to use Hosted Graphite we were more than happy to choose it”.
The result
We now have an excellent understanding of what’s going on inside our systems, both from an infrastructure and application perspective.
Since migrating to Hosted Graphite for their monitoring, Itai and the Big Data group have a much deeper insight into what’s happening with their apps and infrastructure. They also rely heavily on Hosted Graphite’s advanced alerting and receive notifications via Slack to stay on top of unusual activity. This allows the team to track when a threshold is violated and act quickly to rectify issues before Nielsen’s customers are affected.
The team also use the metrics dashboard feature to see how much traffic they’re sending, how it varies over time and across protocols, and the metric treemap features to explore which parts of their infrastructure are generating a lot of metrics and are cost drivers. Hosted Graphite’s continuous improvement has been the cherry on the cake for Itai: “It’s great to know that Hosted Graphite are constantly improving the product and adding new features.”
Hosted Graphite’s transparent pricing–no surprises and no hidden fees–has removed the team’s worry about spiralling costs. According to Itai, “there’s complete transparency with everything Hosted Graphite does, which means we can accurately predict what we’ll be spending and comfortably keep within our budget.”
