Migrating to TimescaleDB

Introduction

Here at MetricFire we’re moving some huge rocks to get more benefits for our customers. Our tech team is migrating our Graphite backend from our original database to TimescaleDB. This will drive huge benefits for our customers stemming from the new ability to access their database through PostgreSQL querying.

Simultaneously, we’ll be migrating our cloud provider from Hetzner to AWS. This drives further benefits surrounding latency, uptime and security requirements for our customers. 

MetricFire has released developments for other major projects, such as our Hosted Prometheus service, which brings the ability to monitor more data sources and gain greater observability. 

Our TimescaleDB migration will be first available in beta. TimescaleDB Beta will be released by the end of Q4, 2020. We’ll be reaching out to customers to get them involved in the Beta version!

Please reach out to us if you have any questions, or if you’re interested in being a part of the beta testing for our TimescaleDB migration! Book a demo with our team and we can talk directly on video chat. 

Now, let’s dive into our latest updates!

Migrating from Hetzner to AWS

Here at MetricFire, we’re changing how our backend is configured so we can provide better service for our customers. The backend for the Hosted Graphite application was originally hosted on Hetzner. Hetzner has been a great home for Hosted Graphite ever since incorporation, however we’re looking to give more flexibility to our customers, especially in terms of data hosting locations. 

MetricFire will be moving our cloud service provider to AWS to improve latency, get more hosting locations, and to get greater uptime.

The benefits of AWS

Having your data hosted in AWS will give you the following benefits:

1. Choice of country for data hosting - by being able to choose the country you want your data hosted in, you will be able to meet security requirements, as well as improve overall latency and experience when interacting with your data.
2. Better latency - AWS is generally considered a stronger and more reliable cloud service provider compared with Hetzner. This will give our customers a better experience with latency when querying. Also, being able to choose the location of your data hosting will provide better latency.
3. Better uptime - AWS will provide generally improved uptime for accessing your data.

Migrating to TimescaleDB

Simultaneously, we will build out a TimescaleDB on our AWS servers rather than continue to use our original database infrastructure. This will enable customers to benefit from everything a PostgreSQL database can give. Check out the benefits below.

The benefits of TimescaleDB

Having your data hosted in TimescaleDB will give you the following benefits:

1. An open database - You will be able to query your database with PostgreSQL and easily find the information you’re looking for. With the existing Hosted Graphite backend, you are only able to see your data via graphs and dashboards. It wasn’t possible to query and apply data directly. With TimescaleDB you’ll be able to query and apply your data for uses such as time-series forecasting.
2. Easy to hire employees - A PostgreSQL database is a very familiar tool among all developers, and it will become easy to hire and find employees who are good at using it.
3. Cost-savings through the easy-identification of unneeded metrics - If your database is open and accessible, you’ll easily be able to identify which metrics are no longer necessary. You can eliminate anything that isn’t applicable anymore, and stop paying for it.

By opening up the database it will allow for us to use the information in the database for better applications. For example, we’ll be able to run machine learning algorithms over the database so that customers can do better anomaly detection.

MetricFire 2.0

The AWS migration, as well as the TimescaleDB migration will pave the way for new features and development at MetricFire.

TimescaleDB is a very popular technology with many connectors available for Python, or other languages, all built by the community. This means that you can easily apply interesting technology, such as time-series forecasting. 

TimescaleDB’s built in connectors enable you to write your own scripts to transform and apply data how you like. As well, chances are the TimescaleDB community has already done development similar to what you’re looking for, and you can pull that technology into your solution.

MetricFire will keep developing so that using your data will become easier and easier. In the release of MetricFire 2.0 in 2021 Q1, we’ll be providing customers new features directly stemming from these new found abilities. Some of these new features will be anomaly detection and time-series forecasting.

The TimescaleDB and AWS migration is in line with MetricFire’s mandate - bringing you the best of the open source world with none of the hassle. TimescaleDB is a great piece of open source software that we want our customers to benefit from. Now you can have it without having to lift a finger!

The old infrastructure specs

Previously MetricFire was running on a modified NoSQL key-value datastore on Hetzner. Moving forward, all customers will continue to be hosted on this original infrastructure and Hetzner. Migrating to AWS/TimescaleDB will be opt-in, with the first few customers working together with us by using the beta version. 

You’ll hear more from us soon about when and how you can get involved!

What this means for Prometheus

MetricFire’s Hosted Prometheus is already hosted completely in AWS, with a Prometheus server for time-series data storage. The migration to AWS and TimescaleDB doesn’t affect our Hosted Prometheus customers as their data stores will remain exactly the same. 

Conclusion

MetricFire is always growing, and we’re excited to announce these new projects! Moving forward we’ll be able to provide even better service to our customers, allowing much better data control and creative applications.

You should book a demo with the MetricFire team if you’re interested to learn more! We'll help you try out the new features, and help you maximize your MetricFire account!