Install Graphite: Common Issues

Install Graphite: Common Issues

Table of Contents

  1. Introduction
  2. First steps in setting up Graphite
    1. Using SQLite3
    2. Automatic start up
    3. Log rotation
  3. Troubleshooting Graphite Installation
  4. Hosted Graphite vs Self Hosted Graphite
    1. Out of the box scaling
    2. Automatic backup of data
    3. 24 / 7 on-call Support
    4. Security
    5. Automatic Updates
  5. Conclusion

Introduction

Graphite is a very popular enterprise monitoring tool. This article will address the common issues that occur while setting up a Graphite instance, and how they can be avoided. We will assume readers have already become acquainted with Graphite, but if you’re interested to learn about the basics of Graphite, check out our articles on the Architecture and Concepts of Graphite and the Installation and Setup before reading this article. For this article, the key terms you need to know are:

  • Carbon - It is the storage server for Graphite. Graphite can run multiple instances of Carbon on a single node to handle the incoming load.
  • Whisper - is the name of a database format which Graphite uses to store the data.

We will also evaluate the benefits and costs of setting up your own instance of Graphite vs using a Hosted Graphite service such as MetricFire. As operations scale, it can become too time-consuming to run your own Graphite instance, making a hosted version much more feasible. 

As you’re going through this article, try out your own Hosted Graphite with our free trial account. You will be able to send metrics directly and test to see how it works. 

  

 

First steps in setting up Graphite

In our installation tutorial, we used docker images of Graphite to run and demo the Graphite application. However, to achieve the best performance, Graphite needs to be installed on a production machine running either CentOS / Debian. This requires following a lot of manual steps to get the Graphite up and running. 

So, let’s get started. In this example, I will be using Ubuntu 18.04 to get the Graphite up and running. 

  1. Update the local package by running the command:

 

sudo apt-get update

 

  1. Install the Graphite web application and Graphite Carbon:

 

sudo apt-get install graphite-web graphite-carbon

 

During the installation process, the prompt will appear asking if you want to keep the Carbon database if you ever decide to uninstall and re-install. You can choose no for this prompt.

 

undefined

  

Although technically now you have installed Graphite and Carbon, there are various caveats which we are going to address now one by one.

 

Using SQLite3

The Graphite web application uses SQLite3 database files by default to store its data. This is not at all recommended for production systems since SQLite3 is not a full fledged database system. Hence, we will install and configure PostGreSQL to let Graphite store the data there.

 

sudo apt-get install postgresql libpq-dev python-psycopg2

 

This will install PostGreSQL to which Graphite will connect to and store the data.

Now, we will create the user Graphite and database Graphite in the database.

 

madhur@madhur-Virtual-Machine:~$ sudo -u postgres psql
psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
Type "help" for help.

postgres=# create user graphite with password 'password';
CREATE ROLE
postgres=# create database graphite with owner graphite;
CREATE DATABASE
postgres=# \q

  

Now, we need to tell Graphite to use our new database server settings instead of the default database which is SQLite. Change the values in /etc/graphite/local_settings.py from the following default values to the new database settings we just created.

Old Values:

‍ 

DATABASES = {
    'default': {
        'NAME': '/var/lib/graphite/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}

 ‍

New Values:

‍ 

DATABASES = {
    'default': {
        'NAME': 'graphite',
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'USER': 'graphite',
        'PASSWORD': 'password',
        'HOST': '127.0.0.1',
        'PORT': ''
    }
}

 

Run the command “sudo graphite-manage migrate” to let Django create tables in the new database. You should see the following output:

 

madhur@madhur-Virtual-Machine:~$ sudo graphite-manage migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, tagging
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying sessions.0001_initial... OK
  Applying tagging.0001_initial... OK
  Applying tagging.0002_on_delete... OK
madhur@madhur-Virtual-Machine:~$ 

 
 

Automatic start up

If your production server gets restarted for some reason, the Graphite and Carbon won’t start automatically. For it to start automatically, we need to make some changes in /etc/default/graphite-carbon: Change the value of CARBON_CACHE_ENABLED to true.

This will ensure that Graphite and Carbon are started automatically on boot.

  

Log rotation

By default, carbon doesn’t do any log rotation. That means, the hard disk is at risk of filling up unless you clean your log files manually or enable log rotation.

 

1. Edit the file /etc/carbon/carbon.conf

2. Change the value of ENABLE_LOGROTATION to True

To get the graphite web interface up and running, we need to install apache

 

sudo apt-get install apache2 libapache2-mod-wsgi

 

To enable the graphite on Apache web server, run the following commands

 

madhur@madhur-Virtual-Machine:~$ sudo a2dissite 000-default
Site 000-default disabled.
To activate the new configuration, you need to run:
  systemctl reload apache2
madhur@madhur-Virtual-Machine:~$ sudo cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available/
madhur@madhur-Virtual-Machine:~$ sudo a2ensite apache2-graphite
Enabling site apache2-graphite.
To activate the new configuration, you need to run:
  systemctl reload apache2
madhur@madhur-Virtual-Machine:~$ sudo service apache2 reload

‍  

Once this is done, if you go to http://localhost, you should finally see Graphite Web application:

 

undefined

 

Troubleshooting Graphite Installation

If you face any issue while setting up Graphite and Carbon for production:

  • Make sure you are using the supported operating systems such as RHEL, CentOS, Debian and Ubuntu.
  • Check to make sure the correct versions of Python, Django, Apache web server and Database engine are being used. Most of the issues arise due to incompatibility between these running parts.
  • Look for existing issues on the following repositories:

https://github.com/graphite-project/carbon/issues

https://github.com/graphite-project/graphite-web/issues

Most of the time, the problem you are facing has been faced by someone else previously as well.

 

Hosted Graphite vs Self Hosted Graphite

If you have made it to the end of article and are able to successfully run Graphite, congratulations. If you face any issue, there are numerous troubleshooting articles on the Internet to get it up and running, which we will not replicate here. You can also reach out to MetricFire and see if a hosted version of Graphite is better suited

In the rest of the article, we will analyze the benefits of using Hosted Graphite such as MetricFire vs using your own Graphite. MetricFire helps in the following ways:

Out of the box scaling

In the article above, we haven't really touched scaling Graphite, which will be required at any production grade deployment. It requires running multiple instances of Carbon in the same host and also across multiple hosts in a cluster. Setting that up isn’t trivial and requires a lot of administrative manpower and monitoring.

 

Automatic backup of data

Hosted Graphite offers automatic backup of data to make sure if something goes wrong, data can be recovered quickly and business isn’t affected. Deploying a custom backup solution in self-hosted Graphite is an expensive exercise and requires deploying additional manpower and storage to take care of backups.

  

24 / 7 on-call Support

Hosted Graphite provides year round support and automated monitoring anywhere in the world.

 

Security

Hosted graphite provides high security using protocols such as HTTPS and TLS to transmit data packets. Configuring self-hosted Graphite to provide such security requires a lot of custom configuration to get it up and running.

 

Automatic Updates

Graphite consists of a lot of moving parts such as Python, Django, Apache Web Server and database engine. In order to run it at the optimum levels, all these components need to be updated at regular intervals. Hosted Graphite solves these problems for you by applying the latest updates for these components in the background.

As well the actual technology of Hosted Graphite is slightly different than regular Graphite. The developers have added to the Graphite project to make Hosted Graphite even easier for the users. We outline this topic in the article Hosted Graphite isn't Graphite.

 

Conclusion

We have seen the process of setting up self hosted Graphite. It doesn’t cover advanced use cases like scaling, security and backups. Such features are only provided by hosted Graphite. Self hosted Graphite requires administration and specialized engineers who are experts in Linux Administration, Graphite, python, Django, Apache Web Server and PostGreSQL. For most setups, the cost of running self hosted Graphite exceeds the cost of using hosted Graphite. 

To try Hosted Graphite, log in to MetricFire's Hosted Graphite free trial, and start sending metrics today. You can also reach out to us by booking a demo, we're always happy to jump on a call and talk about your company's monitoring needs.

Hungry for more knowledge?

Related posts