Monitoring Amazon Streams with PagerDuty

To integrate Amazon Streams and PagerDuty with your monitoring system, please reach out to MetricFire. Book a demo with the MetricFire team to discuss integrating Amazon Streams and PagerDuty and how that can support your monitoring system.

Amazon Kinesis Data Streams (KDS) is a data streaming service that is both scalable and durable and operates in real time. It can capture gigabytes of data per second from hundreds of thousands of sources, including websites, databases, social media feeds, IT logs, financial transactions, and location-tracking events. Real-time analytics tools such as dashboards, anomaly detection, and dynamic pricing are updated in milliseconds.

KDS is based on a producer/consumer model. A producer loads data records into Amazon KDS. A consumer then processes the data records from the stream. For example, a producer could load web server log data to a Kinesis data stream, where a consumer can process and analyze the log data.

You can use Kinesis Data Streams to collect log and event data from servers, desktops, and mobile devices. You can then build Kinesis Applications to process your data continuously, generate metrics, populate dashboards, and aggregate data into stores such as Amazon S3.

You can create Kinesis Applications that run real-time analytics on high-frequency event data events. For example, you could write an application to analyze sensor data collected by Kinesis Data Streams to gain insights into your sensor data.

If you collect gaming data, Kinesis Data Streams can collect data about player-game interactions and feed the data back into your gaming platform. This strategy can help you design a game that provides engaging and dynamic experiences based on players' past actions and behaviors.

With Metricfire, you can turbocharge your data stream monitoring services. By integrating Amazon CloudWatch with the Metricfire platform, you can display your KDS metrics on aesthetically pleasing dashboards. MetricFire's advanced filtering lets you choose only the data views you want to see and discard the rest. You can also set up simple rules to discard data you no longer keep, plus receive alerts via email or Slack when your data streams are doing something they shouldn't.

To get started, first, create a policy that we will later attach to the user.

1.  In the IAM Dashboard, click on Policies (on the left) or Customer Managed Policies under IAM Resources, then click on Create Policy.
2. Switch to the JSON tab, replace the existing text with the policy provided below, and click ‘Review policy‘.
3. Give the policy a name (Eg: HG_policy) and a description (optional).
4. Save the policy by clicking ‘Create Policy‘.

Next, we’ll create a user to attach the policy. We’ll use the Access Key/Secret Key tokens to permit Hosted Graphite to import CloudWatch metric data.

1.  In the IAM Console, click on ‘Users‘ on the left or under IAM Resources. Then select ‘Add User‘.
2. Give the user a name (Eg: HG_addon) and ‘Programmatic access‘ for the Access type.
3. Click ‘Next: Permissions‘ and click the ‘Attach existing policies directly‘ button at the top.
4. Search for the policy we just created and attach it by clicking the corresponding checkbox, then press ‘Next: Review‘.
5. Click on ‘Create User‘ and copy the Access Key and Secret Key into the relevant fields on the add-on set-up page here. Optionally, you can download a copy of this account information, but keep it safe as this information can be sensitive.

PagerDuty is an incident management platform used to help teams quickly detect and fix infrastructure problems. It manages automatic notifications when a problem occurs, escalates the issue where necessary, and schedules on-call resources.

With its SaaS platform, PagerDuty provides businesses, developers, and administrators a range of powerful tools to help them quickly manage client-impacting issues. By arming your business with the insight to proactively manage events that may affect customers across your IT infrastructure, PagerDuty helps you to maintain customer experience and satisfaction. It also allows you to sustain your brand reputation and revenue streams.

PagerDuty comes with hundreds of native integration points to ensure that the right data gets in the hands of the right people in real-time. By taking advantage of machine-learning, PagerDuty automatically handles resource scheduling and issue escalations. You can customize PagerDuty notifications to use a combination of phone calls, SMS, push notifications, and emails depending on the issue severity, time of day, or defined support hours.

These features are why PagerDuty can be such a useful companion to your MetricFire applications such as Hosted Graphite. For example, you might use Hosted Graphite to receive alerts about system performance, code bugs, or other issues. If you use PagerDuty as well, you would find it of immense use when you wanted automatic delegating to the people in your organization. One way would be to use PagerDuty to create alerts that it automatically assigns to specific people on your team.

You can create a PagerDuty service on MetricFire's Hosted Graphite that triggers once an alert criterion reaches a specified threshold. Once triggered, the service delegates tasks to specific individuals, leaving you free to focus on other important job functions.

To integrate Amazon Streams and PagerDuty with your monitoring system, sign up for a free trial with MetricFire. Talk with the MetricFire team about how to integrate Amazon Streams and PagerDuty and get Amazon Streams and PagerDuty interacting with your MetricFire dashboards directly.

MetricFire is a full-scale platform that provides infrastructure, system, and application monitoring using a suite of open-source monitoring tools. We enable you to use Hosted Graphite and aesthetic custom dashboards to visualize your metrics so you can understand what is happening.

MetricFire offers a complete ecosystem of end-to-end infrastructure monitoring, comprised of open-source Graphite and popular dashboards. Plugins for many other open-source projects are preconfigured, such as StatsD, collectd, and more. Not only does MetricFire fit well into many monitoring use cases, such as server monitoring, but we also offer application and cloud infrastructure monitoring.

Through this hosted environment, MetricFire boosts the unique features of open-source projects to give you more functionality than the original products. Below are some of the MetricFire features at a glance: 

• Cloud Monitoring
  
  
• Hosted Graphite
  
  
• Dashboards

The key thing to remember is that Hosted Graphite by MetricFire is more than just Graphite. Our Hosted Graphite product actually adds data dimensionality and better data storage.

Benefits of Using MetricFire:

• Simple, low-cost pricing
  A structured pricing model based on unique time series metrics allows you to work within your budget. The predictability and transparent pricing allow you to keep your costs in check and plan for the future. 1 metric is 1 metric.
  
  
  
• Easy-to-use dashboards
  Easily share your dashboards with clients for free.
  
  
  
• Responsive alerting
  Alert notification integrations PagerDuty, Slack, email, and webhooks.
  
  
  
• Freedom of customization
  Custom metrics through your code.
  
  
  
• Fantastic customer support
  Highly available support is provided by engineers for engineers to get you set up quickly.
  
  
  
• Enterprise-ready
  Dedicated clusters for users that need their own environment.