Monitoring Amazon CloudWatch with Amazon EMR

To integrate Amazon CloudWatch and Amazon EMR with your monitoring system, please reach out to MetricFire. Book a demo with the MetricFire team to discuss integrating Amazon CloudWatch and Amazon EMR and how that can support your monitoring system.

Amazon CloudWatch is a management and monitoring service designed for AWS and other infrastructure resources or on-premises applications. It is the official metrics monitoring tool for Amazon Web Services. Using CloudWatch, you can access all your performance and operational metrics in a single platform, helping you overcome the challenge of monitoring multiple systems. CloudWatch helps you monitor your entire stack — including applications, infrastructure, and services — thus freeing up valuable resources to allow you to focus on building applications.

You can use CloudWatch Container Insights to monitor and troubleshoot your applications and microservices. CloudWatch collects, aggregates, and summarizes computer utilization information; like CPU and memory usage, network data history, and also monitoring diagnostic information. Container Insights provides you with details about container management services, such as Amazon ECS for Kubernetes (EKS), Amazon's Elastic Container Service (ECS), etc.

The brilliant thing about Amazon Cloudwatch is that it is your gatekeeper to data and metrics for all your Amazon applications and services. However, monitoring more than the standard set of metrics can become very expensive with CloudWatch. CloudWatch custom metrics are very expensive and they should be used sparingly. For example, if a company is monitoring its AWS systems with the standard CloudWatch dashboards, it might cost around 1000 USD a month. However, if you’re monitoring hundreds of thousands of metrics related to a new launch, AWS CloudWatch could quickly rack up to 50,000 USD a month.

That's why it's such a vital integration point for MetricFire. MetricFire treats all metrics the same, so if you’re monitoring thousands of specialized metrics, you’ll still pay the same basic rate for those metrics. CloudWatch can be integrated with MetricFire, so you can pull your AWS metrics into the MetricFire platform. Then, you can get low-cost metrics scaling, while still being able to monitor your AWS metrics all in a single pane of glass. MetricFire's advanced filtering lets you choose only the data views you want to see and discard the rest. You can also set up simple rules to discard data you no longer need to keep, plus receive alerts via email or Slack.

To get started, first, create a policy that we will later attach to the user.

1.  In the IAM Dashboard, click on Policies (on the left) or Customer Managed Policies under IAM Resources, then click on Create Policy.
2. Switch to the JSON tab, replace the existing text with the policy provided below, and click ‘Review policy‘.
3. Give the policy a name (Eg: HG_policy) and a description (optional).
4. Save the policy by clicking ‘Create Policy‘.

Next, we’ll create a user to attach the policy. We’ll use the Access Key/Secret Key tokens to permit Hosted Graphite to import CloudWatch metric data.

1.  In the IAM Console, click on ‘Users‘ on the left or under IAM Resources. Then select ‘Add User‘.
2. Give the user a name (Eg: HG_addon) and ‘Programmatic access‘ for the Access type.
3. Click ‘Next: Permissions‘ and click the ‘Attach existing policies directly‘ button at the top.
4. Search for the policy we just created and attach it by clicking the corresponding checkbox, then press ‘Next: Review‘.
5. Click on ‘Create User‘ and copy the Access Key and Secret Key into the relevant fields on the add-on set-up page here. Optionally, you can download a copy of this account information, but keep it safe as this information can be sensitive.

To enable the CloudWatch add-on, go to the add-ons page in your Hosted Graphite account and choose the option for Amazon AWS CloudWatch. From there you can select the AWS services you wish to connect with.

 

 

 

  

If you have any questions about getting your AWS instances connected to Hosted Graphite, contact our team. We're happy to help you!

MetricFire is a full-scale platform that provides infrastructure, system, and application monitoring using a suite of open-source tools. We will aggregate and store your data as time series metrics, which can be used to build custom dashboards and alerts. MetricFire takes away the burden of self-hosting your own monitoring solution, allowing you more time and freedom to work on your most important tasks.

MetricFire offers a complete ecosystem of end-to-end infrastructure monitoring, comprised of open-source Graphite and Grafana. MetricFire handles the aggregation, storage, and backups of your data, and offers alerting, team features, and API's for easy management of your monitoring environment. You can send server metrics using one of our agents, custom metrics from within your application code, and integration metrics from a variety of popular 3rd party services that we integrate with like Heroku, AWS, Azure, GCP, and many more!

Our Hosted Graphite product has improved upon standard Graphite to add data dimensionality, optimized storage, and offers additional tools and features that provide customers with a robust and well-rounded monitoring solution.

Benefits of Using MetricFire:

• Simple, low-cost pricing
  A structured pricing model based on total unique time series metrics allows you to work within your budget. The predictability and transparent pricing allows you to keep your costs in check and plan for the future. 1 metric is 1 metric regardless of the source, or requests sent to and from the metric namespace.
  
• Easy-to-use dashboards
  Our Hosted Dashboards provide endless options for customizing your visualizations, and allow you to share dashboards with clients.
  
• Responsive alerting
  Receive alert notifications quickly through Email, PagerDuty, Slack, Microsoft Teams, OpsGenie, and custom webhooks.
  
• Freedom of customization
  Send custom application metrics through your code. Our Language Guide can help you update your codebase to send event metrics via socket connections.
  
• Fantastic customer support
  Fast and friendly support is provided by engineers, for engineers, to get you set up quickly. Start a conversation with us through the chat bubble below!
  
• Enterprise-ready
  Dedicated clusters for users that need their own environment with custom parameters. Please reach out to us for more information.