Using Telegraf to send syslog metrics to Graphite

Introduction

When you own and operate software, they generate various types of logs from disparate sources such as databases, servers, and applications. The metrics from these important digital assets are what companies monitor continuously. When they show you a sign of unreliability, companies need to take swift actions to fit the cause and prevent it from growing to a larger problem. The key to success in this activity is owning a good Syslog application and metrics software where you can clearly see metrics. This article discusses how we can use Telegraf - a Syslog software - to send Syslog metrics to Graphite. 

      

Before we begin, check out MetricFire. MetricFire offers comprehensive monitoring solutions with hosted software such as Hosted Graphite. You can gain an in-depth insight into your IT resources with minimal configuration. If you want to learn more, Book a demo with us, or sign up for a two-week free trial today.

         

What is Graphite?

Graphite is a tool that allows you to collect and save your system performance metrics. It has basic tools to graphically display stored metrics. Alternatively, you can connect Graphite to other more advanced data visualization systems such as Grafana.

            

What is Telegraf?

Telegraf is a monitoring client that you can easily install and configure. It works with various operating systems (Windows, MacOS, Linux, Red Hat, and CentOS) and has a Graphite output plugin. Installing and configuring Telegraf is different depending on your operating system. In spite of the differences, they share the following steps in general:

         

1. Install Telegraf.
2. Create a configuration file.
3. Make special settings in the Graphite section of the generated file.
4. Launch Telegraf.

     

Let’s go through each step in more detail.

      

Install Telegraf

Depending on your OS, installing Telegraf requires a different process. If you use Windows, you can run the following commands in Powershell.

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.21.4_windows_amd64.zip -UseBasicParsing -OutFile telegraf-1.21.4_windows_amd64.zip
Expand-Archive .\telegraf-1.21.4_windows_amd64.zip -DestinationPath 'C:\Program Files\InfluxData\telegraf'

       

The download file above is the latest stable version as of February 2022. If you want to find a later version, check the Telegraf download page.

       

If you use MacOS, you can run the command below.

brew update
brew install telegraf

      

On Linux, you can execute the commands.

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.21.4_linux_amd64.tar.gz
tar xf telegraf-1.21.4_linux_amd64.tar.gz

      

After you install Telegraf, you need to configure it to send Syslog metrics to Graphite.

       

Create a configuration file

Creating a configuration file also follows different steps depending on your OS.    

• Windows: run ./telegraf.exe --output-filter graphite config > telegraf.conf 
• MacOS: find the configuration file at /usr/local/etc/telegraf/telegraf.conf 
• Linux: find the configuration file at /etc/telegraf/telegraf.conf

         

Modify config to send metrics to Hosted Graphite

The main mechanism for sending metrics to Graphite is via TCP protocol. To send metrics, you need to change the config variables according to your settings within the configuration file you created or found. Below is a sample configuration: 

## This is the configuration for Graphite server to send metrics to
[[outputs.graphite]]
  ## TCP endpoint for your graphite instance.
  ## If multiple endpoints are configured, the output will be load balanced.
  ## Only one of the endpoints will be written to with each iteration.
  ## You can put your hosted graphite endpoint URL
  servers = ["carbon.hostedgraphite.com:2003"]
  ## Metrics prefix name
  prefix = "YOUR_API_KEY.telegraf"
  ## Graphite output template
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
  template = "host.tags.measurement.field"

  ## Graphite templates patterns
  ## 1. Template for cpu
  ## 2. Template for disk*
  ## 3. Default template
  # templates = [
  #  "cpu tags.measurement.host.field",
  #  "disk* measurement.field",
  #  "host.measurement.tags.field"
  #]

  ## Enable Graphite tags support
  # graphite_tag_support = false

  ## Metric name and field separating character for Graphite tag
  # graphite_separator = "."

  ## timeout in seconds for the write connection to Graphite
  timeout = 2

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"

  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

       

Within the configuration above, you need to mainly check the following three areas:     

• [[outputs.graphite]]
• The server URL in the servers list field: carbon.hostedgraphite.com:2003
• Put your prefix following the pattern of "YOUR_API_KEY.telegraf".

       

Save the configuration file after changing the values. Next, you need to restart the Telegraf service to apply the changes. 

        

Launch Telegraf

As the final step, you need to launch Telegraf to apply the changes. The starting command differs from the operating system.

      

Windows

./telegraf.exe -service install -config 'C:\Program Files\telegraf\telegraf.conf'
net start telegraf

      

MacOS

brew services restart telegraf

brew services restart telegraf

      

Linux

sudo service telegraf start

     

or

sudo systemctl start telegraf

           

When Telegraf sends out data, it can decide the output data format following your configuration. 

            

                  

Building a Grafana dashboard in a hosted environment

When you interpret data, visualization is an essential component since people can comprehend via pictures much faster. Since you now set up the pipeline between Telegraf and Graphite, the final piece will be displaying the metrics on dashboards. 

             

Grafana is an open-source monitoring tool that allows you to download data from different sources and visualize it on beautiful, customizable dashboards. After creating a Grafana dashboard, you need to add information panels to it. There are different types of panels in Grafana that you can use for different purposes. For example:     

• Graph Panel. Time charts display data points on the time axis. You can compare different indicators on the same chart, and use the timeshift functions to compare current values ​​with any past time period.
• Single stat. Used to display values ​​resulting from a query or display texts using value mapping transformation options.
• Gauge takes one input series and displays the value in terms of the position it occupies within predefined lower and upper bounds.
• Polystat panel displays small hexagons or circles representing single or composite indicators and their enabled states.
• The Table provides discrete key data for metrics, and supports multiple modes for time series, annotations, and JSON data.
• The Alert list allows you to display all triggered alerts from the control panel independently from different panels.

          

These convenient features can be a great addition to your analytics stack. However, due to the nature of open-source software, you will have to manage everything from installation to maintenance, which can be a burden on your data team. MetricFire provides a managed service for Grafana. You can easily create an account with a pre-configured Grafana instance and start using it right away along with Hosted Graphite.

    

If you want to test how to monitor your system’s performance metrics using Grafana, try the MetricFire free trial or book a demo with our team.

       

The advantages of MetricFire

MetricFire offers a hosted Graphite that you can use to collect and store your system’s performance metrics. MetricFire configures, installs, and maintains Graphite, while you can use it as a web application without having to worry about its performance.

           

Key benefits of using hosted Graphite from MetricFire:

• Data availability: MetricFire provides permanent access to your data, which you can export at any time.
• Affordable price: There are various tariff plans, among which you will definitely choose the one that suits you.
• Saving time and money: There is no need to invest in deploying and running monitoring systems.
• Reliable support: We are always happy to answer all questions regarding working with MetricFire.
• Redundancy storage: Graphite’s default storage is file-based and antiquated. MetricFire offers 3 times the redundant storage for seamless scaling and better data protection.
• Control by APIs: The APIs that MetricFire provides lets you control and automate the resources of Hosted Graphite.
• Tagged metrics: Hosted Graphite stores data using tags that enable viewing and organizing metrics with data views.

      

Hosted Graphite keeps all the benefits of open-source Graphite and further enhances the tool with the built-in agent, team accounts, granular dashboard permissions, and integrations to other major platforms and services such as AWS, Heroku, logging tools, and more.

       

Conclusion

We looked at why it is important to monitor system performance metrics, how to send the collected metrics to Graphite, and how to visualize them using Grafana. Also, we listed the main benefits of using a Hosted Graphite solution from MetricFire. Telegraf is one of the most popular Syslog applications. When it is integrated into Graphite and its metric data is visualized on Grafana, you can fully utilize the powerful features of Telegraf.

              

If you still have any questions, we will definitely answer them. Simply book a demo with the MetricFire team or sign up for the MetricFire for free.