Install Graphite: Common Issues

March 26, 2020

Introduction

Graphite is a very popular enterprise monitoring tool. This article will address the common issues that occur while setting up a Graphite instance, and how they can be avoided. We will assume readers have already become acquainted with Graphite, but if you’re interested to learn about the basics of Graphite, check out our articles on the Architecture and Concepts of Graphite and the Installation and Setup before reading this article. For this article, the key terms you need to know are:

  • Carbon - It is the storage server for Graphite. Graphite can run multiple instances of Carbon on a single node to handle the incoming load.
  • Whisper - is the name of a database format which Graphite uses to store the data.


We will also evaluate the benefits and costs of setting up your own instance of Graphite vs using a Hosted Graphite service such as MetricFire. As operations scale, it can become too time-consuming to run your own Graphite instance, making a hosted version much more feasible. 

As you’re going through this article, try out your own Hosted Graphite with our free trial account. You will be able to send metrics directly and test to see how it works. 


First steps in setting up Graphite

In our installation tutorial, we used docker images of Graphite to run and demo the Graphite application. However, to achieve the best performance, Graphite needs to be installed on a production machine running either CentOS / Debian. This requires following a lot of manual steps to get the Graphite up and running. 

So, let’s get started. In this example, I will be using Ubuntu 18.04 to get the Graphite up and running. 

  1. Update the local package by running the command:

sudo apt-get update


  1. Install the Graphite web application and Graphite Carbon:

sudo apt-get install graphite-web graphite-carbon

During the installation process, the prompt will appear asking if you want to keep the Carbon database if you ever decide to uninstall and re-install. You can choose no for this prompt.



Although technically now you have installed Graphite and Carbon, there are various caveats which we are going to address now one by one.


Issue #1 - Using SQLite3

The Graphite web application uses SQLite3 database files by default to store its data. This is not at all recommended for production systems since SQLite3 is not a full fledged database system. Hence, we will install and configure PostGreSQL to let Graphite store the data there.

sudo apt-get install postgresql libpq-dev python-psycopg2


This will install PostGreSQL to which Graphite will connect to and store the data.

Now, we will create the user Graphite and database Graphite in the database.


<p>CODE:https://gist.github.com/denshirenji/835a8571b2db79c0763b60fa6003af64.js</p>


Now, we need to tell Graphite to use our new database server settings instead of the default database which is SQLite. Change the values in /etc/graphite/local_settings.py from the following default values to the new database settings we just created.


Old Values:

<p>CODE:https://gist.github.com/denshirenji/5331b79d4d36f78f58e1cdafa8c4f600.js</p>

New Values:

<p>CODE:https://gist.github.com/denshirenji/01a5e13bf62d868496ee1eafb99d52c9.js</p>

Run the command “sudo graphite-manage migrate” to let Django create tables in the new database. You should see the following output:


<p>CODE:https://gist.github.com/denshirenji/ee377251268588ffef253da82665c17f.js</p>



Issue #2 - Automatic start up


If your production server gets restarted for some reason, the Graphite and Carbon won’t start automatically. For it to start automatically, we need to make some changes in /etc/default/graphite-carbon: Change the value of CARBON_CACHE_ENABLED to true.

This will ensure that Graphite and Carbon are started automatically on boot.


Issue #3 - Log rotation

By default, carbon doesn’t do any log rotation. That means, the hard disk is at risk of filling up unless you clean your log files manually or enable log rotation.


1. Edit the file /etc/carbon/carbon.conf

2. Change the value of ENABLE_LOGROTATION to True


To get the graphite web interface up and running, we need to install apache

sudo apt-get install apache2 libapache2-mod-wsgi


To enable the graphite on Apache web server, run the following commands

<p>CODE:https://gist.github.com/denshirenji/46e21e668dd898f355e2fcd3017f37d2.js</p>

Once this is done, if you go to http://localhost, you should finally see Graphite Web application:



Troubleshooting Graphite Installation

If you face any issue while setting up Graphite and Carbon for production:

  • Make sure you are using the supported operating systems such as RHEL, CentOS, Debian and Ubuntu.
  • Check to make sure the correct versions of Python, Django, Apache web server and Database engine are being used. Most of the issues arise due to incompatibility between these running parts.
  • Look for existing issues on the following repositories:

https://github.com/graphite-project/carbon/issues

https://github.com/graphite-project/graphite-web/issues


Most of the time, the problem you are facing has been faced by someone else previously as well.


Hosted Graphite vs Self Hosted Graphite

If you have made it to the end of article and are able to successfully run Graphite, congratulations. If you face any issue, there are numerous troubleshooting articles on the Internet to get it up and running, which we will not replicate here. You can also reach out to MetricFire and see if a hosted version of Graphite is better suited

In the rest of the article, we will analyze the benefits of using Hosted Graphite such as MetricFire vs using your own Graphite. MetricFire helps in the following ways:


  • Out of the box scaling

In the article above, we haven't really touched scaling Graphite, which will be required at any production grade deployment. It requires running multiple instances of Carbon in the same host and also across multiple hosts in a cluster. Setting that up isn’t trivial and requires a lot of administrative manpower and monitoring.


  • Automatic backup of data

Hosted Graphite offers automatic backup of data to make sure if something goes wrong, data can be recovered quickly and business isn’t affected. Deploying a custom backup solution in self-hosted Graphite is an expensive exercise and requires deploying additional manpower and storage to take care of backups.


  • 24 / 7 on-call Support

Hosted Graphite provides year round support and automated monitoring anywhere in the world.


  • Security

Hosted graphite provides high security using protocols such as HTTPS and TLS to transmit data packets. Configuring self-hosted Graphite to provide such security requires a lot of custom configuration to get it up and running.


  • Automatic Updates

Graphite consists of a lot of moving parts such as Python, Django, Apache Web Server and database engine. In order to run it at the optimum levels, all these components need to be updated at regular intervals. Hosted Graphite solves these problems for you by applying the latest updates for these components in the background.


As well the actual technology of Hosted Graphite is slightly different than regular Graphite. The developers have added to the Graphite project to make Hosted Graphite even easier for the users. We outline this topic in the article Hosted Graphite isn't Graphite.


Conclusion

We have seen the process of setting up self hosted Graphite. It doesn’t cover advanced use cases like scaling, security and backups. Such features are only provided by hosted Graphite. Self hosted Graphite requires administration and specialized engineers who are experts in Linux Administration, Graphite, python, Django, Apache Web Server and PostGreSQL. For most setups, the cost of running self hosted Graphite exceeds the cost of using hosted Graphite. 

To try Hosted Graphite, log in to MetricFire's Hosted Graphite free trial, and start sending metrics today. You can also reach out to us by booking a demo, we're always happy to jump on a call and talk about your company's monitoring needs.

Related Posts

GET FREE MONITORING FOR 14 DAYS