Monitoring Kubernetes with Graphite

May 8, 2020

Introduction

In this article, we will be covering how to monitor Kubernetes using Graphite, and we’ll do the visualization with Grafana. The focus will be on monitoring and plotting essential metrics for monitoring Kubernetes clusters. We will download, implement and monitor with the custom dashboards for Kubernetes that can be downloaded from the Grafana dashboard resources. These monitoring dashboards have variables to allow drilling down into data at granular level.

To follow along with this blog, sign up for the MetricFire free trial, where you can use Graphite and Grafana directly in our platform. MetricFire is a hosted Graphite, Grafana and Prometheus service, where we do the setup and management of these open-source tools so you don’t have to. 

Introduction to Kubernetes

According to Kubernetes, Kubernetes is a “portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.”

In today’s industry, monitoring Kubernetes deployments is a challenging but necessary activity. In this article, we are not going to cover Kubernetes basics, as we’ll jump more into how to use the dashboards. For a more detailed introduction to Kubernetes monitoring, refer to our Introduction to Monitoring Kubernetes on the MetricFire blog. 


Introduction to Graphite

Graphite is a very popular enterprise monitoring tool, and here at MetricFire we have a Hosted Graphite product that has been developing and improving since 2012. Graphite is a time-series monitoring tool that receives metrics data pushed to it by a collector such as StatsD or collectd, and then monitors the time-series metrics as specified by the user. If you’re interested to learn about the basics of Graphite, check out our articles on the Architecture and Concepts and the Installation and Setup of Graphite before reading this article. 


Setting up Graphite, Grafana and Kubernetes

For the purpose of this article, we will use a Kubernetes cluster deployed on AWS. Follow the instructions on the Amazon AWS EKS user guide to install AWS CLI, and make sure you also install the kubectl command line tool.

Once the kubectl is installed, running the command “kubectl cluster-info” should give the following output:

<p>CODE:https://gist.github.com/denshirenji/47fa4dac5af14ca2574192af5ff337c5.js</p>


Next, let’s set up Grafana on our kubernetes cluster. Here is the simple configuration file which will create a Grafana Pod and a Service running on Kubernetes:

<p>CODE:https://gist.github.com/denshirenji/94112262be8a99f7bb920a27f5c33590.js</p>


Run, “kubectl create -f grafana-deployment.yml” to create Grafana Pod and Service.

If we run the command “kubectl get service”, you should get an output similar to below:

<p>CODE:https://gist.github.com/denshirenji/2ae1f4e4952101b473be8e7bdf04d1d7.js</p>


a942a31a4780f11eaa9010a814a720da-1449083553.ap-south-1.elb.amazonaws.com is the newly created service hostname and 3000 is the port to access. Browse to http://a942a31a4780f11eaa9010a814a720da-1449083553.ap-south-1.elb.amazonaws.com:3000 and login using admin/admin to make sure Grafana is up and running, as shown below:



Similarly, let’s create Pod and Service for Graphite. We will use the below configuration file to create Graphite service.

<p>CODE:https://gist.github.com/denshirenji/c5778168b49eef9cb1b7d04d2d8645a2.js</p>


Run, “kubectl create -f graphite-deployment.yml” to create this Pod and Service.

At this point, both Grafana and Graphite, should be up and running. 

Run, “kubectl get service” to make sure both the services are up and running.

<p>CODE:https://gist.github.com/denshirenji/927d0f1701ffcaddc53d1526a4afbe41.js</p>


Just like Grafana, we can enter http://ac0f466207b2211eaa9010a814a720da-687823427.ap-south-1.elb.amazonaws.com in the browser to open the Graphite web application, as shown below:


Now, we will add Graphite as the data source in Grafana. Browse to the data source section of Grafana and add Graphite as the data source, as shown below:


Now, we will run Snap Daemon in our Kubernetes cluster. Snap Daemon is the monitoring daemon which will pull various kubernetes monitoring metrics and push them into Graphite.

Before we run the Snap Daemon, we need to make one small change inside snap_ds.yml. We will update the hostname and port of Graphite service in the config section as shown below:

<p>CODE:https://gist.github.com/denshirenji/9e27f271c80f2b40e2e5a0debb38315c.js</p>


Run, “kubectl create -f snap_ds.yml” to create Snap Daemon and get it started. If we now run, “kubectl get pod”, we will see output similar to below

<p>CODE:https://gist.github.com/denshirenji/f4b0507de14b6eb6978172fe8732fbb1.js</p>


We see one Pod for Grafana, one for Graphite and three for Snap. This is because we have 3 nodes in our Kubernetes cluster and Snap will run as a daemon on each of the nodes to pull its metrics and push it to Graphite.

Plotting the monitoring visualization on Grafana

Grafana comes with lots of pre-built dashboards that you can find on the Grafana dashboards resources site. We are going to utilize one of these dashboards for our Kubernetes monitoring: Kubernetes Container Stats

On the Kubernetes Container Stats page, click the link “Download JSON” and import it into our Grafana portal. Make sure to choose the Graphite data source appropriately.



As soon as the dashboard is imported, we should see the metrics being shown in our dashboard, similar to below:





Similarly, there is another dashboard Kubernetes Node host metrics. When this is imported, it will show the metrics per host selected in the dashboard. 



You might want to set up alerts on these dashboards if the values in these dashboards exceed some critical threshold. Check out our article Grafana Dashboards from Basic to Advanced to learn how to set up Grafana alerts, and build custom dashboards.

You can also create other types of visualizations based on the metrics exposed by Kubernetes. Have a look at the article Our Favorite Grafana Dashboards to create some of the more advanced dashboards.


Setting up the Monitoring through MetricFire

The setup which we have done above works for very basic Kubernetes infrastructure which would contain just a few nodes. In order to handle production level load, which would be a few hundred nodes and upwards of a few Mbps network traffic, you would need to scale out both Graphite and Grafana to handle the increasing load. 

That’s where Hosted Graphite and Hosted Grafana comes into the picture. It allows you to scale for long-term storage, as well as automatically providing redundant storage of data. 

Hosted Graphite and Hosted Grafana through MetricFire allows for the continuous active deployment of new features, as MetricFire’s products all have their foundations in the ever-growing open source projects. Sign up for the MetricFire free trial here, and start building Kubernetes dashboards within a few minutes.


Conclusion

In this article, we have seen how to set up Kubernetes monitoring with Graphite. We have seen some advanced visualizations to monitor Kubernetes using Graphite and Grafana. 

Sign up here for a free trial of our Hosted Graphite and Grafana offering. Also, if you have any questions about our products, or about how MetricFire can help your company, talk to us directly by booking a demo

Related Posts

GET FREE MONITORING FOR 14 DAYS