Monitoring Amazon Lambda with Amazon Firehose

To integrate Amazon Lambda and Amazon Firehose with your monitoring system, please reach out to MetricFire. Book a demo with the MetricFire team to discuss integrating Amazon Lambda and Amazon Firehose and how that can support your monitoring system.

Amazon Lambda is a cloud-based, fully scalable service that lets you run code without managing or provisioning servers. Amazon Lambda will run your code when you need it and only charge you when you use it. You can run virtually any code for nearly all types of backend applications and services with no administration. Amazon looks after it all for you, including code monitoring and logging, capacity provision and scaling, and operating system and server maintenance.

You can use code that responds automatically to an Amazon DynamoDB table or Amazon's Simple Storage Service (Amazon S3). You can also invoke code using API calls made using AWS SDKs or write code to respond to HTTP requests using Amazon API Gateway. Plus, you can write functions for serverless applications that you can automatically deploy using AWS CodeBuild and AWS CodePipeline. You can create backend services that take advantage of Amazon Web Services' inbuilt security, scale, and performance, plus use Amazon Lambda to extend other AWS services with custom logic.

More importantly, Amazon Lambda automatically monitors your code on your behalf and reports metrics through Amazon CloudWatch. Amazon Lambda automatically tracks the number of requests, the invocations per request, and the number of errors triggered by a request. 

With MetricFire, you can turbocharge your Amazon Lambda code monitoring services. By integrating Amazon CloudWatch with the MetricFire platform, you can display your Amazon Lambda metrics on aesthetically pleasing dashboards. MetricFire's advanced filtering lets you choose only the data views you want to see and discard the rest. You can also set up simple rules to discard data you no longer keep, plus you can receive alerts via email or Slack when your code is doing something it shouldn't.

To get started, first, create a policy that we will later attach to the user.

1.  In the IAM Dashboard, click on Policies (on the left) or Customer Managed Policies under IAM Resources, then click on Create Policy.
2. Switch to the JSON tab, replace the existing text with the policy provided below, and click ‘Review policy‘.
3. Give the policy a name (Eg: HG_policy) and a description (optional).
4. Save the policy by clicking ‘Create Policy‘.

Next, we’ll create a user to attach the policy. We’ll use the Access Key/Secret Key tokens to permit Hosted Graphite to import CloudWatch metric data.

1.  In the IAM Console, click on ‘Users‘ on the left or under IAM Resources. Then select ‘Add User‘.
2. Give the user a name (Eg: HG_addon) and ‘Programmatic access‘ for the Access type.
3. Click ‘Next: Permissions‘ and click the ‘Attach existing policies directly‘ button at the top.
4. Search for the policy we just created and attach it by clicking the corresponding checkbox, then press ‘Next: Review‘.
5. Click on ‘Create User‘ and copy the Access Key and Secret Key into the relevant fields on the add-on set-up page here. Optionally, you can download a copy of this account information, but keep it safe as this information can be sensitive.

With the ever-increasing popularity of streaming content, the need to enable users to collect, store, capture, and process large amounts of data from their distributed streams is crucial. By using Amazon Kinesis Data Firehose, you can capture massive amounts of live video data from millions of sources, and then make them available to consumers in a data-efficient and cost-effective format.

Amazon Kinesis Data Firehose lets you quickly load streaming data into data stores, data lakes, and analytics services. You can use it to capture, transform, and deliver streaming data to a range of Amazon services, including Amazon Redshift, Amazon S3, and Amazon Elasticsearch Service. You can also use Kinesis to connect to generic HTTP endpoints and service providers such as New Relic, MongoDB, Datadog, and Splunk.

Firehose is a fully managed service that automatically scales to match your data throughput. You can also use it to compress, transform, and encrypt your data streams. This feature minimizes the amount of storage required for your streaming data and increases security.

Amazon Kinesis Data Firehose integrates with Amazon CloudWatch metrics so you can collect and analyze your data streaming metrics. For example, you could monitor the IncomingBytes and IncomingRecords metrics to track data ingested into Kinesis Data Firehose from your data producers. You can use Kinesis Analytics to analyze and process your data streams using standard SQL. It also comes with pre-built stream processing templates to help you manage your data analytics.

By integrating Amazon CloudWatch with the MetricFire platform, you can display your streaming metrics on aesthetically pleasing dashboards. MetricFire's advanced filtering lets you choose only the data views you want to see and discard the rest. You can also set up simple rules to discard data you no longer need, plus receive alerts via email or Slack when your data streaming is doing something it shouldn't.

To get started, first, create a policy that we will later attach to the user.

1.  In the IAM Dashboard, click on Policies (on the left) or Customer Managed Policies under IAM Resources, then click on Create Policy.
2. Switch to the JSON tab, replace the existing text with the policy provided below, and click ‘Review policy‘.
3. Give the policy a name (Eg: HG_policy) and a description (optional).
4. Save the policy by clicking ‘Create Policy‘.

Next, we’ll create a user to attach the policy. We’ll use the Access Key/Secret Key tokens to permit Hosted Graphite to import CloudWatch metric data.

1.  In the IAM Console, click on ‘Users‘ on the left or under IAM Resources. Then select ‘Add User‘.
2. Give the user a name (Eg: HG_addon) and ‘Programmatic access‘ for the Access type.
3. Click ‘Next: Permissions‘ and click the ‘Attach existing policies directly‘ button at the top.
4. Search for the policy we just created and attach it by clicking the corresponding checkbox, then press ‘Next: Review‘.
5. Click on ‘Create User‘ and copy the Access Key and Secret Key into the relevant fields on the add-on set-up page here. Optionally, you can download a copy of this account information, but keep it safe as this information can be sensitive.

To integrate Amazon Lambda and Amazon Firehose with your monitoring system, sign up for a free trial with MetricFire. Talk with the MetricFire team about how to integrate Amazon Lambda and Amazon Firehose and get Amazon Lambda and Amazon Firehose interacting with your MetricFire dashboards directly.

MetricFire is a full-scale platform that provides infrastructure, system, and application monitoring using a suite of open-source monitoring tools. We enable you to use Hosted Graphite and aesthetic custom dashboards to visualize your metrics so you can understand what is happening.

MetricFire offers a complete ecosystem of end-to-end infrastructure monitoring, comprised of open-source Graphite and popular dashboards. Plugins for many other open-source projects are preconfigured, such as StatsD, collectd, and more. Not only does MetricFire fit well into many monitoring use cases, such as server monitoring, but we also offer application and cloud infrastructure monitoring.

Through this hosted environment, MetricFire boosts the unique features of open-source projects to give you more functionality than the original products. Below are some of the MetricFire features at a glance: 

• Cloud Monitoring
  
  
• Hosted Graphite
  
  
• Dashboards

The key thing to remember is that Hosted Graphite by MetricFire is more than just Graphite. Our Hosted Graphite product actually adds data dimensionality and better data storage.

Benefits of Using MetricFire:

• Simple, low-cost pricing
  A structured pricing model based on unique time series metrics allows you to work within your budget. The predictability and transparent pricing allow you to keep your costs in check and plan for the future. 1 metric is 1 metric.
  
  
  
• Easy-to-use dashboards
  Easily share your dashboards with clients for free.
  
  
  
• Responsive alerting
  Alert notification integrations PagerDuty, Slack, email, and webhooks.
  
  
  
• Freedom of customization
  Custom metrics through your code.
  
  
  
• Fantastic customer support
  Highly available support is provided by engineers for engineers to get you set up quickly.
  
  
  
• Enterprise-ready
  Dedicated clusters for users that need their own environment.