Monitoring Amazon SQS with Amazon ECS

To integrate Amazon SQS and Amazon ECS with your monitoring system, please reach out to MetricFire. Book a demo with the MetricFire team to discuss integrating Amazon SQS and Amazon ECS and how that can support your monitoring system.

Amazon Simple Queue Service (SQS) is a message queuing service that you use to decouple and scale microservices, distributed systems, and serverless applications. Amazon SQS supports programmatic sending of messages via web service applications as a way to communicate over the Internet. It is fully managed and eliminates the complexity and overhead associated with operating message-oriented middleware. Amazon SQS is used by companies such as Netflix, Dropbox, NASA, and more.

With SQS, you can send, store, and receive messages between software components without the need for another service. You can get started with SQS in minutes using the AWS console, Command-Line Interface, or SDK of your choice.

SQS offers two types of message queues:

• Standard queues provide maximum throughput, best-effort ordering, and at least once delivery. 
• SQS FIFO queues ensure messages are processed exactly once, in the exact order you send them.

You can use Amazon SQS to add an extra layer of security to your messaging services. You can exchange sensitive data between applications using server-side encryption (SSE) to encrypt each message. Plus, because Amazon SQS scales elastically with your applications, it is a highly cost-effective messaging solution.

Amazon SQS is fully integrated with Amazon CloudWatch, allowing you to view and analyze your Amazon SQS queue metrics, which is where MetricFire comes into play.

With Metricfire, you can turbocharge your Amazon SQS monitoring services. By integrating Amazon CloudWatch with the Metricfire platform, you can display your metrics on aesthetically pleasing dashboards. MetricFire's advanced filtering lets you choose only the data views you want to see and discard the rest. You can also set up simple rules to discard data you no longer keep, plus receive alerts via email or Slack when your message queues are doing something they shouldn't.

To get started, first, create a policy that we will later attach to the user.

1.  In the IAM Dashboard, click on Policies (on the left) or Customer Managed Policies under IAM Resources, then click on Create Policy.
2. Switch to the JSON tab, replace the existing text with the policy provided below, and click ‘Review policy‘.
3. Give the policy a name (Eg: HG_policy) and a description (optional).
4. Save the policy by clicking ‘Create Policy‘.

Next, we’ll create a user to attach the policy. We’ll use the Access Key/Secret Key tokens to permit Hosted Graphite to import CloudWatch metric data.

1.  In the IAM Console, click on ‘Users‘ on the left or under IAM Resources. Then select ‘Add User‘.
2. Give the user a name (Eg: HG_addon) and ‘Programmatic access‘ for the Access type.
3. Click ‘Next: Permissions‘ and click the ‘Attach existing policies directly‘ button at the top.
4. Search for the policy we just created and attach it by clicking the corresponding checkbox, then press ‘Next: Review‘.
5. Click on ‘Create User‘ and copy the Access Key and Secret Key into the relevant fields on the add-on set-up page here. Optionally, you can download a copy of this account information, but keep it safe as this information can be sensitive.

Amazon Elastic Container Service (Amazon ECS) is a container management service that is fully managed, fast, and secure. It makes it easy for you to run, stop, and manage containers on a cluster. It should not be confused with Amazon EC2 which is used to manage the computing capacity and resources of the infrastructure used to store and run your containers. However, Amazon ECS and Amazon EC2 are often used together to manage your containers and the infrastructure they run on.

Containers are used for packaging application code, configurations, and dependencies into a single object, ensuring quick, reliable, and consistent deployments, regardless of your software environment. AWS provides a range of tools to help you register, manage, and run your application containers.

You can create task definitions that you use to run individual tasks or tasks within a service for each of your containers. AWS Fargate can run your tasks for you, or if you need more control, you can run your services or tasks on a cluster of Amazon EC2 instances.

Amazon ECS launches and stops your container-based applications by using simple API calls. You can also retrieve your cluster's state from a centralized service that gives you access to many Amazon EC2 features.

Monitoring your Amazon ECS resources is simple by using Amazon CloudWatch. The metrics you collect depend on the task launch type you use. If you use Fargate launch types for your services, then CPU and memory utilization metrics are provided to monitor your services. For the Amazon EC2 launch types, you need to monitor the EC2 instances yourself. This is where MetricFire can help you out.

With MetricFire, you can turbocharge your Amazon ECS monitoring services. By integrating Amazon CloudWatch with the MetricFire platform, you can display your metrics on aesthetically pleasing dashboards. MetricFire's advanced filtering lets you choose only the data views you want to see and discard the rest. You can also set up simple rules to discard data you no longer keep, plus receive alerts via email or Slack when your ECS service is doing something it shouldn't.

To get started, first, create a policy that we will later attach to the user.

1.  In the IAM Dashboard, click on Policies (on the left) or Customer Managed Policies under IAM Resources, then click on Create Policy.
2. Switch to the JSON tab, replace the existing text with the policy provided below, and click ‘Review policy‘.
3. Give the policy a name (Eg: HG_policy) and a description (optional).
4. Save the policy by clicking ‘Create Policy‘.

Next, we’ll create a user to attach the policy. We’ll use the Access Key/Secret Key tokens to permit Hosted Graphite to import CloudWatch metric data.

1.  In the IAM Console, click on ‘Users‘ on the left or under IAM Resources. Then select ‘Add User‘.
2. Give the user a name (Eg: HG_addon) and ‘Programmatic access‘ for the Access type.
3. Click ‘Next: Permissions‘ and click the ‘Attach existing policies directly‘ button at the top.
4. Search for the policy we just created and attach it by clicking the corresponding checkbox, then press ‘Next: Review‘.
5. Click on ‘Create User‘ and copy the Access Key and Secret Key into the relevant fields on the add-on set-up page here. Optionally, you can download a copy of this account information, but keep it safe as this information can be sensitive.

To integrate Amazon SQS and Amazon ECS with your monitoring system, sign up for a free trial with MetricFire. Talk with the MetricFire team about how to integrate Amazon SQS and Amazon ECS and get Amazon SQS and Amazon ECS interacting with your MetricFire dashboards directly.

MetricFire is a full-scale platform that provides infrastructure, system, and application monitoring using a suite of open-source monitoring tools. We enable you to use Hosted Graphite and aesthetic custom dashboards to visualize your metrics so you can understand what is happening.

MetricFire offers users a complete ecosystem of end-to-end infrastructure monitoring, comprised of popular open-source monitoring software services: Graphite and popular dashboards. Plugins for many other open-source projects are preconfigured, such as StatsD, collectd, and Kubernetes. You get all these within a hosted environment as a single product. Not only does MetricFire fit well into the infrastructure monitoring use-case, such as network monitoring and server monitoring, but we also do application monitoring and business intelligence.

Through this hosted environment, MetricFire boosts the unique features of open-source projects to give you more functionality than the original products. Below are some of the MetricFire features at a glance: 

• Cloud Monitoring
  
  
• Hosted Graphite
  
  
• Dashboards

The key thing to remember is that Hosted Graphite by MetricFire is more than just Graphite. Our Hosted Graphite product actually adds data dimensionality and better data storage.

The benefits of MetricFire are:

• Simple, low-cost pricing
  A structured pricing model based on unique time series metrics allows you to work within your budget. The predictability and transparent pricing allow you to keep your costs in check and plan for the future. 1 metric is 1 metric.
  
  
  
• Easy-to-use dashboards
  Easily share your dashboards with clients for free.
  
  
  
• Responsive alerting
  Alert notification integrations PagerDuty, Slack, email, and webhooks.
  
  
  
• Freedom of customization
  Custom metrics through your code.
  
  
  
• Fantastic customer support
  Highly available support is provided by engineers for engineers to get you set up quickly.
  
  
  
• Enterprise-ready
  Dedicated clusters for users that need their own environment.