Monitoring Docker Containers with cAdvisor

November 1, 2019


Docker is one of the most popular tools for containerization, and several tools have been developed by the open source community to monitor what happens inside of Docker containers. This guide focuses on one tool specifically: cAdvisor.


Why do we need to monitor Docker containers?

Monitoring allows us to gather vital information on the state of our software, enabling development teams to figure out ways in which to improve on their product. Because each container is a self-contained system, monitoring is even more imperative.  


Monitoring for a container involves recording and reporting usage metrics from each container back to the monitoring system. This way, you ensure detection of bugs and application refinement for better all-around performance and robustness.


How to use cAdvisor for container monitoring

cAdvisor (Container Advisor) is an open source container monitoring tool offered and managed by Google. It has native support for Docker containers and just about any other container. cAdvisor consists of a single container daemon that collects information about the containers that are running, processes that data, and then exports it. This information can go to its dedicated web interface, or to a third party app, such as Big Query, ElasticSearch, InfluxDB, Kafka, Prometheus, Redis, or StatsD. It can also be written to standard output by setting the -storage_driver flag to stdout. 


Specifically, cAdvisor records historical resource usage, resource isolation parameters and network statistics for each container machine-wide. 


Quick start: running cAdvisor  

Setting up and running cAdvisor is a simple task because cAdvisor itself is a single Docker image. As such, it can be started and run using the docker run command. You can run cAdvisor to monitor containers on your local machine by running the following command in your terminal: 


sudo docker run \

  --volume=/:/rootfs:ro \

  --volume=/var/run:/var/run:ro \

  --volume=/sys:/sys:ro \

  --volume=/var/lib/docker/:/var/lib/docker:ro \

  --volume=/dev/disk/:/dev/disk:ro \

  --publish=8080:8080 \

  --detach=true \

  --name=cadvisor \

  google/cadvisor:latest

It’s that easy! cAdvisor is now running in the background. It will connect itself to the Docker daemon running locally, gather the container usage metrics and display them on its web user interface accessible via http://localhost:8080/. On the user interface, you should see real time usage metrics from your containers including CPU and memory usage, a detailed overview of all your running processes, and so much more. 


Notes:

  • If your docker daemon is running with user namespace enabled, you will need to add the  --userns=host option to enable cAdvisor to monitor Docker containers. Otherwise, cAdvisor will be unable to connect to the Docker daemon.
  • If you’re running on either CentOS, Fedora or RHEL, you may need to run the container with --privileged=true and --volume=/cgroup:/cgroup:ro \ in order for cAdvisor to access and monitor Docker containers. This is because RHEL and CentOS are stricter when it comes to container access. With cAdvisor needing access to the Docker daemon through its socket, you will have to set --privileged=true. Also, in some versions of RHEL and CentOS the cgroup hierarchies are mounted in the /cgroup directory, so you will need the additional Docker option of --volume=/cgroup:/cgroup:ro \ to run cAdvisor.


Collecting Application Metrics with cAdvisor

In addition to container usage metrics, cAdvisor is also capable of collecting application metrics, including the number of active and reading connections, and whether or not the application has adequate CPU and memory allocation. (It is worth noting that, at the time of writing, application metrics support by cAdvisor is in the alpha stage of development, so keep that in mind.) A container can expose application metrics on a status page or on a separate API for fetching stats. cAdvisor provides a way to collect these metrics in a generic manner.


In order to enable cAdvisor to collect application metrics, there are two things you need to do:

  • Create a configuration
  • Pass that configuration to cAdvisor


Creating a configuration

An application metric configuration tells cAdvisor where to search for application metrics, and then specifies other parameters to export metrics from cAdvisor to user interfaces and backends. The metric configuration includes:

  • An endpoint (a location to collect metrics from)
  • Type (gauge, counter, etc.)
  • Name of metric
  • Data type (float, int)
  • Polling frequency
  • Units (seconds, kbps, count)
  • Regexps (regular expressions to specify which metrics to collect and how to parse and handle them)

Below is an example of a generic metric collector which assumes no structured information:

{

  "endpoint" : "http://localhost:8000/nginx_status",

  "metrics_config" : [

    {

      "name" : "activeConnections",

      "metric_type" : "gauge",

      "units" : "number of active connections",

      "data_type" : "int",

      "polling_frequency" : 10,

      "regex" : "Active connections: ([0-9]+)"

    },

    {

      "name" : "reading",

      "metric_type" : "gauge",

      "units" : "number of reading connections",

      "data_type" : "int",

      "polling_frequency" : 10,

      "regex" : "Reading: ([0-9]+) .*"

    }

  ]

}


In the case of structured metrics exports to Prometheus, for example, the configuration can be shrunk to just the endpoint (other information can be obtained from the structure). Here is a sample Prometheus configuration that collects all metrics from an endpoint:

{

  "endpoint" : "http://localhost:9100/metrics"

}

Here’s another sample configuration that collects only selected metrics:

{

  "endpoint" : "http://localhost:8000/metrics",

  "metrics_config" : [

    "scheduler_binding_latency",

    "scheduler_e2e_scheduling_latency",

    "scheduling_algorithm_latency"

  ]

}

Passing the configuration to cAdvisor

cAdvisor uses Docker container labels to fetch configurations for each Docker container. Any label beginning with io.cadvisor.metric is parsed as a cAdvisor application metric label. cAdvisor uses the value of that label as an indicator of where the configuration can be found. For example, labels of the form io.cadvisor.metric.prometheus-xyz suggest that the configuration points to a Prometheus metrics endpoint.


The configuration file can either be a part of the container image or can be added later on (at runtime) with a volume. This ensures that there is no connection between the host  within which the container is running, and the application metrics configuration. A container’s metric information is self-contained, so a sample configuration for Redis would look like this: 

Dockerfile (or runtime):

FROM redis

 ADD redis_config.json /var/cadvisor/redis_config.json

 LABEL io.cadvisor.metric.redis="/var/cadvisor/redis_config.json"


Where redis_config.json is the configuration file that contains the json configurations as shown above. cAdvisor will then reach into the container image at runtime, process the configuration file, and start collecting and exposing application metrics.

Note that cAdvisor specifically looks at the container labels to extract this information. In Docker 1.8, containers don't inherit labels from their images, and thus you must specify the label at runtime.

API access to application-specific metrics

You can access application-specific metrics for a particular container using the following endpoint:  http://localhost:8080/api/v2.0/appmetrics/containerName

 

The set of application-metrics being collected can be discovered from the container specifications:  http://localhost:8080/api/v2.0/spec/containerName

Regular stats API also has application-metrics appended to it:   http://localhost:8080/api/v2.0/stats/containerName

Note: You can find the application-metrics on the container page after the resource metrics.


cAdvisor, however, has its limitations. It is unable to send alerts back to the user to provide them with critical information. For example, if one of the measured metrics hits a critical level, cAdvisor wouldn’t inform you. Take this into consideration when using it. 


Regardless, cAdvisor is still an excellent tool for container monitoring due to its exceptionally easy setup and seamless Docker support. If you really need to incorporate an alerting mechanism, there are many options you can choose from including Prometheus and StatsD. We’ll walk through a quick example with Prometheus below.


Monitoring and Exporting cAdvisor Metrics with Prometheus


cAdvisor is capable of  exposing container statistics to Prometheus out of the box. By default, these metrics are served under the /metrics HTTP endpoint. This endpoint can be customized by setting the -prometheus_endpoint command-line flag to the desired value.


To monitor cAdvisor with Prometheus, we have to configure one or more jobs in Prometheus which scrape the relevant cAdvisor processes at that metrics endpoint. 


But first, we’ll go ahead and configure Prometheus. We do this using the prometheus.yml file. Create a prometheus.yml file and paste into it the code below (the code is from my GitHub page):


<p>CODE:https://gist.github.com/tenas97/89db0ebd369f257df22f4924f06ff1da.js</p>


The global variables can be explained below:

  • scrape_interval: how frequently targets are scraped for metrics (the default is one minute)
  • scrape_timeout: the time it takes for a scrape request to time out (the default is 10 seconds)
  • rule_files: path to rule files, in this case, the alert rule file which specifies how we handle alerts when they come up
  • evaluation_interval: how frequently to evaluate rules (the default is 1 minute)


In the same folder where you created the prometheus.yml file, create a docker-compose.yml file and populate it with this Docker Compose configuration (also from my GitHub page):



<p>CODE:https://gist.github.com/tenas97/9129a4a3b58cd0853bb6f00fdd84144b.js</p>

Now we create our alert.rules file which specifies how to handle a specific alert. Below is an example of a simple alert.rules file available in my GitHub page. 

<p>CODE:https://gist.github.com/tenas97/4829c24c2afef1df57edd3afc8a06d4f.js</p>


This alert is triggered when a specific container (“your container name”) is stopped and absent for 5 seconds or more. Alerts in Prometheus are handled by the Prometheus Alert Manager, which is pretty flexible and allows for alerts to be instantly routed to an application of your choice, be it an email, a slack workspace, or more. In order to set this up, take a look at the alert manager configuration documentation.


And there you have it! You have successfully monitored a Docker container (or many!), can see its usage and host metrics using cAdvisor, and can receive alerts when a condition which you specify is met. Pretty cool!


Related Posts

GET FREE monitoring FOR 14 Days