Securing your monitoring infrastructure

Securing Your Monitoring Infrastructure

Table of Contents

Introduction

Your monitoring system provides a comprehensive overview of any infrastructure. To effectively monitor your infrastructure and systems, you’ll need to get all of your data into one place - regardless if you have 1 node or 10 nodes. This centralization of data inevitably creates a vulnerable point that attackers can potentially target and exploit.

In this article, we look at how to design your infrastructure in a secure way, as well as focus in on how to secure your nodes. In the second half of this article, we will exclusively focus on the central node that is the point of aggregation between data that is taken from internal nodes. 

By working with MetricFire, you can hand over the worry of maintaining your monitoring to the experts, and ensure that your monitoring infrastructure is fully secured. Check out our free trial, and book a demo to talk about monitoring security! 

   

Where are the Vulnerabilities

There are a few critical ways that a monitoring system can be exploited.

First, in the case where sensitive data is accessed in an unauthorized way (such as during an attack), a nefarious party may have the ability to modify, alter, and/or manipulate the information in your dashboards as an effort to hide possible red flags.

Second, monitoring an infrastructure requires a specific set of skills and if you are not an expert, it is quite easy to make errors that can lead to misconfigurations. As we can see from a previous article, Monitoring your own infrastructure with open-source Graphite and Grafana, configuring Graphite and Grafana is not a simple task. 

Potential errors are sometimes difficult to detect due to the complex nature of the configurations, and misconfigurations or other errors can become an open door into your infrastructure for attackers.

     

How to Secure These Vulnerabilities

In the face of a threat, given the variability of an infrastructure, there is no exact way to manage such an attack; rather, there are only practical and logical behaviors and methods to put into practice. 

This article is not intended to be a complete guide, but it provides insight that can be used to deepen your research within the area of computer security. 

Now, we will take a look at the best common practices for securing your monitoring systems. 

   

The Zero Trust Architecture

It is considered best practice to implement a zero trust architecture, where every component (including internal components) “doubts” the trust of the user (hence, zero trust). 

This setup ensures that there is no default trust of any entity and further ensures that an attacker will have difficulty executing on its own attack given the compartmentalization of the architecture. While configuring a zero trust architecture is resource intensive, it is considered one of the best infrastructures that can be included in a thoughtfully designed IT security plan.

     

Limit Your Infrastructure’s Exposure

The best way to reduce your attack surface is to limit your infrastructure’s exposure on the web. As with most companies, there is intranet (internal company network) and extranet (the company network facing outwards). 

In this context, two elements are useful for us: firewall and reverse proxy. In the following tutorial sections, we assume that the nodes are set to dialogue only with the internal company network.

The following tutorial sections are running on a machine with the IP x.x.x.x and domain http://example.com with a Debian distribution installed.

   

         

Installing the Firewall

As previously mentioned, we will exclusively focus on the central node that is the point of aggregation between data that is taken from internal nodes. In this section, let’s install the firewall on the central node.

On all Unix based systems, there are tables that contain some rules for packet filtering. Each TCP/IP packet contains a series of information that is useful for iptables to decide whether or not to accept a packet. In a very simple way, this is how a firewall works. Although iptables is the most complete solution, it is often complicated to configure for newbies.

UFW (Uncomplicated Firewall) is a particular utility developed in Python that allows for the simple configuration of iptables. It is one of the best utilities developed by the Linux community and, in some instances, you can find it installed by default on some distributions, such as Ubuntu.

To install UFW, we proceed with updating the package repository:

root@nodo apt update
root@nodo apt install ufw

     

If all went well, call ufw -v from shell should return the usage version. UFW is based on using and enforcing rules to block or accept requests on ports. The classic syntax is as follows:

ufw [option] [port/protocol]

  • option: can be ALLOW or DENY
  • port: interface port (integer value between 0 and 65535, according to TCP/IP standard); it could be combined with a protocol specifying protocol (TCP or UDP)

Our approach will now be to first prohibit all incoming and outgoing connections and then open only those of interest.

ufw default deny incoming
ufw default deny outgoing

       

Done this way, a possible attacker will not be able to enter our node. In the case where the attacker violates this rule by other programs (where the port is whitelisted), the attacker will not be able to open a shell because all connections from the node to the outside are blocked. It is important to note that before enabling the firewall, it is vital to whitelist SSH or the protocol from which you are talking to the machine (RDP, or similar).

ufw allow ssh

        

So, now we are ready to enable the firewall:

ufw enable

      

For the moment, we do not enable any additional rules. To check the status of our firewall (and which ports it blocks), we launch the command:

ufw status

        

Let’s install the reverse proxy and then open the ports on Grafana and Graphite:

ufw allow [port graphite]/tcp
ufw allow [port graphite]/udp

         

Installing NginX

A reverse proxy is a particular server used to forward all external requests made by a client to the internal server, therefore eliminating exposure. This constitutes an additional layer of “layering” as it can be configured together with a load balancer to handle multiple requests without tiring the internal server.

In our case, we will have the internal server that will be used in Grafana, while the reverse proxy will be NginX. NginX is a well known web server (together with Apache) developed in C by Igor Sysoev. NetCraft, a popular internet company, has estimated that approximately 36% of the web is served through NginX.

NginX has the ability to serve as an HTTP server, but also as load balancer and reverse proxy. Unlike Apache, to handle each request, NginX uses an asynchronous event-based approach instead of single threads. Therefore, the consumption resources of NginX is much lower than Apache ― making it a logical choice in this context.

As a first step, we install NginX through package management:

root@node apt update && apt install nginx

        

To verify the success of the installation, we enable port 80 (HTTP) and 443 on the UFW firewall:

root@node ufw allow 'Nginx Full'

      

Now, let’s check http://example.com. If you see “NginX works”, then the installation procedure went well. After this, we insert the reverse proxy. It is important to note that all NginX configurations are located in the directory /etc/nginx/sites-enabled ― copy the default configuration file and enable the reverse proxy.

NginX has a few simple rules for writing the configuration. Each line represents a directive and must end with a semicolon (;). If a line starts with #, then it is a comment. Several directives can be included in a single section called a “block” depending on the service they modify (for example, the “HTTP” block is used for directives that modify or alter the HTTP response of a server).

root@node cp /etc/nginx/sites-enabled/default /etc/nginx/sites-enabled/example.com
root@node nano /etc/nginx/sites-enabled/example.com

        

From here, we modify the HTTP block:

server {
  listen       80;
  server_name  example.com;
location / {
    proxy_pass         127.0.0.1:3000;
    proxy_redirect     off;
    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection "upgrade";
  }
}   

     

The proxy_pass directive is important, as it specifies the internal address to which NginX must connect. The other directives are to set the headers of the client ― the internal server must know to whom it sends the information, otherwise a client would receive the answer of another client.

To verify that the configuration is syntactically correct, we run NginX with the -t flag:

root@nodo nginx -t

      

If there is no error, let’s restart NginX (making sure that the graph-web service is active):

root@nodo systemctl restart nginx

    

Let’s access http://example.com. After doing so, the Grafana user interface should appear.

   

Installing Let’s Encrypt and Obtaining Your First TLS Certificate

The HTTP protocol is one of the major protocols used to communicate through the client-server model. In the client-server model, the client requests a service from the server. The exchange of information between the client and the server is carried out in clear text; although, the information is not encrypted.

This being the case, one of the fundamental properties of computer security is missing, namely the authenticity of the message. With this protocol, there is no way for the client to verify whether the server’s response is actually the original response. To resolve this, a new type of protocol was introduced, TLS (Transportation Layer Security, ex-SSL).

The TLS protocol allows client-server applications to communicate over a network in an effort to prevent data tampering, forgery, and interception. Computer attacks such as “Man In the Middle”, an attack where a malicious node replicates sent packets by modifying them at will, are preventable thanks to TLS.

HTTPS, an extension of HTTP, is an application layer protocol that provides HTTP traffic encryption using TLS. To enable HTTPS in your domain, you need to obtain a certificate that guarantees the authenticity of the message. Let’s Encrypt is a non-profit authority that allows you to obtain free certificates for your infrastructure.

Now, let’s install CertBot ― a utility that enables you to get a certificate in a simple and fast way without being an encryption expert:

sudo apt-get install certbot python-certbot-nginx

        

Let’s generate the new certificate and install it!

sudo certbot --nginx

     

The certificates generated by CertBot are of limited duration (about 90 days), after that you have to renew them through the following command:

sudo certbot renew --dry-run

    

Conclusion: Security Requires Constant Attention

The maintenance of a security system is as important as its design and installation phase. These are complex systems with different technological equipment that must be 100% operational if they are to perform their function. A bug may have the potential to be catastrophic for our monitoring infrastructure and, in turn, could result in unintended consequences for an entire company.

As we have seen with Let’s Encrypt, and the application of the “critical” patches in the previous article, data security requires constant maintenance. It is important to remember that a thorough maintenance design is the result of using best-in-class resources ― for companies, this requires substantial time and money. If a company decides to subcontract these responsibilities to a third party, such a company must be diligent in selecting a third party, as it may be underwhelmed with the services provided.

If you have time constraints and want to work efficiently and effectively, MetricFire allows you to sleep soundly thanks to its 24/7 team that constantly monitors the security of your monitoring infrastructure and applies continuous updates to keep up with the best-in-class standards. Check out MetricFire’s demo or free trial for more!

You might also like other posts...
product Oct 11, 2023 · 17 min read

The Most Important KPIs for Monitoring Mobile Games

This article takes a deep dive into the top KPIs for mobile game monitoring.... Continue Reading

product Oct 06, 2023 · 12 min read

How to Monitor Redis Performance

Use MetricFire to monitor your Redis Performance. See this how-to guide on Redis monitoring... Continue Reading

product Oct 05, 2023 · 13 min read

Best Monitoring Tools for Hadoop

Check out the best monitoring tools for Hadoop and see how to set up... Continue Reading

header image

We strive for
99.999% uptime

Because our system is your system.

14-day trial 14-day trial
No Credit Card Required No Credit Card Required