Table of Contents
Grafana is a popular tool for monitoring and analytics. It allows us to build dashboards to visualize, query, and analyze data, and set up alert notifications for certain conditions. To start from the beginning, check out our article on Grafana Dashboards, and our article called Getting Started with Grafana. This article will go in-depth on the most popular Grafana data sources and how to use them.
Get started by signing up for the MetricFire free trial, where we have a Hosted Grafana Service that you can use without any installation or configuration. This free trial will let you follow along with this article by using Grafana directly in your browser.
Key Takeaways
- Grafana is a powerful tool for monitoring and analytics, enabling the creation of dashboards to visualize, query, and analyze data, along with setting up alert notifications.
- Grafana relies on data sources, which are external locations where data is stored. It doesn't require data to be loaded into Grafana; instead, connections are established between Grafana and data sources to analyze and monitor data.
- To add a data source in Grafana, access the Configuration menu, click on Data Sources, and select the preferred data source. Configure the necessary parameters and save the connection.
- Grafana supports various data sources, including cloud providers like AWS CloudWatch, Azure Monitor, and Google Stackdriver; relational databases like PostgreSQL, MySQL, and Microsoft SQL Server; time series databases like InfluxDB, Prometheus, and Graphite; and others like Elasticsearch, Loki, and Google BigQuery.
What is a data source?
Grafana works with data, which must be stored in a database before it can be accessed by Grafana. There are several different kinds of databases. Additionally, some other systems can be used to store data, even though their main purpose is not focused on data storage.
A Grafana data source is any place from which Grafana can pull data. In other words, you don’t need to load data into Grafana to analyze it. Instead, you should create a connection between Grafana and the data source. Grafana will then work with the connected data source to analyze the data stored there derive some insights and perform monitoring.
How to add a data source in Grafana
To add a data source in Grafana, you should hover over the gear icon at the top right (Also called the Configuration menu) and then click on the Data Sources button:
In the Data Sources section of the Configuration menu, you can press the Add Data Source button. In the screenshot below you can see that we’ve already added data sources such as Hosted Graphite. If you have a fresh local installation of Grafana, you will not see any data sources here until you add them. In general, this is the place where you can view all of your connected data sources.
When you click the Add data sources button, you will see the list of officially supported data sources available for connection:
Select the preferred data source and click on it. The next step will be to specify the required parameters (URL, authorization details, names, etc.):
After you finish filling in the required parameters, you need to press the Save & Test button:
When Grafana establishes the connection, you will see the corresponding message. Then, you can go to the Dashboards section and start exploring the data from the connected data source.
Popular Grafana data sources
Let’s now explore and briefly describe the data sources that are available “out of the box” in Grafana. We can split them into a few key groups:
- Cloud data sources
In this group you will find the monitoring solutions from the huge cloud providers:
AWS CloudWatch
CloudWatch is a tool for application and resource monitoring. It works both with cloud-based solutions and on-premises.
Azure Monitor
Azure Monitor is the monitoring solution from Microsoft. It provides services for tracking applications, infrastructure, and networks.
Google Stackdriver
Stackdriver is a similar monitoring service from Google.
- SQL data sources
In this group, you will see the most popular relational databases.
PostgreSQL
The advanced object-relational database. It provides many features that are not available in other similar instruments. If you have the data with timestamps in PostgreSQL you can use this plugin to visualize it in Grafana.
MySQL
Probably the most popular relational database today. Again, time-series data from MySQL can be effectively visualized in Grafana.
Microsoft SQL Server
MS SQL Server is the “third whale” in the relational databases zoo along with PostgreSQL and MySQL.
- Time series databases
Time series data is the best type of data for monitoring and visualizing in Grafana.
InfluxDB
One of the most popular databases for storing time-series data. InfluxDB is a part of the TICK-stack: the collection of tools for collecting, storing, and analyzing time-series data. Chronograph is the tool from the TICK-stack that plays the same role as Grafana, but Grafana is considered a more flexible and versatile solution.
Prometheus
Prometheus is a well-known time-series database and monitoring tool. Prometheus is a time-series database that includes tagging, monitoring, and querying functionality.
Graphite
Graphite is a convenient monitoring tool. It can be installed on-premise or in the cloud. Graphite’s time-series database is not cluster native and doesn’t have as versatile tagging functionality as Prometheus. However, MetricFire’s Hosted Graphite service has expanded upon the open-source projects and now offers these features.
OpenTSDB
OpenTSDB is a scalable time-series database and monitoring system. - Logging and document databases:
Elasticsearch
Elasticsearch is a powerful search engine that can also be used as the NoSQL database.
Loki
Loki is the log-aggregation system from Grafana.
- Others
There are some other officially supported data sources:
- simplejson
- Logz.io
- TestData DB
Popular community-driven Grafana data sources
Let’s describe some popular community-driven data sources.
Cloudera Manager
Cloudera Manager is the tool for Apache Hadoop administration. It also provides functionality for monitoring and reporting. Using this plugin you can connect the Cloudera Manager to Grafana and build dashboards using data collected from the Hadoop cluster.
Consul
Consul is the solution for connecting cloud services and securing them. This helps move from static to dynamic architecture. Consul collects metrics that reflect the health and performance of the connected services, which you can monitor in Grafana.
Druid
Apache Druid is a real-time analytics database. With the community-supported plugin, you can access the data stored in Apache Druid from Grafana.
Google BigQuery
Google BigQuery is a well-known cloud data warehouse that allows reliable storage of data and efficient analytics.
Google Calendar
This plugin could be used to access information about events scheduled with Google Calendar. For each event, there is the start and end time, that’s how the transformation to the time-series view occurs.
IBM APM
IBM Cloud Application Performance Management enables application monitoring. With this plugin, you can collect and visualize metrics from IBM APM in Grafana.
Oracle Cloud Infrastructure
This plugin allows running queries against the Oracle Cloud Infrastructure Monitoring tool.
ClickHouse
ClickHouse is a popular open-source column-oriented database often used for storing information about events in web analytics, gaming software, etc. Events (for example, user clicks) usually have date-time fields, which makes it possible to transform to time-series data and analyze it in Grafana.
Finance
This plugin is developed for fetching data from Quandl API - a financial data provider. Financial data is usually time-series data, so it can be used easily in Grafana.
DarkSky
DarkSky API provides you with historical information about weather conditions and future forecasts. This plugin allows you to monitor the weather in Grafana, which could be very important for many applications and use cases.
The list above is incomplete. There are many other community-driven data source plugins. In addition, developers constantly create new data sources and make them available for the community. So, if you need a specific data source, it is always a good idea to check the page for community-supported data source plugins.
Installing third-party data sources
As you remember, when you click on the Add data source button, you will be presented with the list of officially supported data sources. If you scroll the page down, you will see the "Find more data source plugins on grafana.com" link:
If you follow this link, you will see the page with the available plugins (you can see the selection menu on the left-hand side, showing the plugin type as data source):
The officially supported plugins are denoted as “by Grafana Labs”, while community-supported plugins show the names of their developers:
If you click on any of the data sources provided here, you will enter the page with the details about the plugin and the installation information:
After you install the plugin, the data source should appear in the list of available data sources in your Grafana UI.
You can find more detailed instructions about plugin installation here.
Creating custom data sources
If you cannot find the needed data source among the officially supported or community-provided ones, you can create your own data source. The idea is that a custom plugin can be created for any database that uses HTTP protocol for communication with clients. The plugin should transform data from the database into time-series data for Grafana to be able to correctly represent it in the dashboard's visualizations.
To develop a plugin for the new data source, three components are required:
- Data source JavaScript object (responsible for communication with the database and transformation of the stored data)
- QueryCtrl JavaScript class (for editing metrics in the dashboards’ panels)
- ConfigCtrl JavaScript class (for performing the configuration of the new data source of this type, or editing it by the user)
Conclusion
This article explored Grafana data sources and explained how to work with them. We described a range of popular officially supported data sources, as well as community-driven plugins. Finally, we introduced the process of installing third-party data sources and creating your own data sources.
If you want to use Grafana for your monitoring or data analysis needs, don’t forget that we have a 14-day free trial for Grafana as a Service. If you have any questions, please contact us or write to sales@metricfire.com.
